Answer category data classifying using dynamic thresholds

ABSTRACT

Managing confidence data in a question-answering environment is disclosed. Managing confidence data can include sorting, based on a set of answer categories for a subject matter, a first set of a plurality of answers into a first answer category. The first set can correspond to at least one of a third set of a plurality of confidence scores and the second set can correspond to at least one of a fourth set of the plurality of confidence scores. Managing confidence data can include classifying confidence scores of the third set into one of a plurality of confidence buckets using a first threshold and determining a fifth set of a plurality of thresholds using the plurality of confidence scores. Managing confidence data can include classifying unclassified confidence scores of the third set into one of the plurality of confidence buckets using the fifth set of the plurality of thresholds.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. Provisional Application No.62/075,635 filed Nov. 5, 2014, entitled “Parameter Management in aQuestion-Answering Environment,” the entirety of which is herebyincorporated herein by reference.

BACKGROUND

The present disclosure relates to answer management in aquestion-answering (QA) environment and, more specifically, toestablishing an answer sequence from the group of answers sortedaccording to a sequence of answer categories.

Question-answering (QA) systems can be designed to receive inputquestions, analyze them, and return applicable answers. Using varioustechniques, QA systems can provide mechanisms for searching corpora(e.g., databases of source items containing relevant content) andanalyzing the corpora to determine answers to an input question.

SUMMARY

According to embodiments of the present disclosure, a method formanaging confidence data in a question-answering environment isdisclosed. The method can include sorting, based on a set of answercategories for a subject matter, a first set of a plurality of answersinto a first answer category and a second set of the plurality ofanswers into a second answer category. Each of the first set cancorrespond to at least one of a third set of a plurality of confidencescores and each of the second set can correspond to at least one of afourth set of the plurality of confidence scores. The plurality ofconfidence scores can represent confidence of answers to a querysubmitted to a question-answering system. The method can includeclassifying confidence scores of the third set into one of a pluralityof confidence buckets using a first threshold and determining a fifthset of a plurality of thresholds using the plurality of confidencescores. The method can include classifying unclassified confidencescores of the third set into one of the plurality of confidence bucketsusing the fifth set of the plurality of thresholds.

Embodiments of the present disclosure are directed to a system formanaging confidence data in a question-answering environment. The systemcan include a processor, and a computer readable storage medium havingprogram instructions embodied therewith.

In embodiments, the program instructions can be executable by theprocessor to cause the system to sort, based on a set of answercategories for a subject matter, a first set of a plurality of answersinto a first answer category and a second set of the plurality ofanswers into a second answer category. Each of the first set cancorrespond to at least one of a third set of a plurality of confidencescores and each of the second set can correspond to at least one of afourth set of the plurality of confidence scores. The plurality ofconfidence scores can represent confidence of answers to a querysubmitted to a question-answering system. The program instructions cancause the system to classify confidence scores of the third set into oneof a plurality of confidence buckets using a first threshold, determinea fifth set of a plurality of thresholds using the plurality ofconfidence scores, and classify unclassified confidence scores of thethird set into one of the plurality of confidence buckets using thefifth set of the plurality of thresholds.

Embodiments of the present disclosure are directed to a computer programproduct for managing confidence data in a question-answeringenvironment. The computer program product comprising a computer readablestorage medium having program instructions embodied therewith, theprogram instructions executable by a computer to cause the computer toperform a method.

The method can include sorting, based on a set of answer categories fora subject matter, a first set of a plurality of answers into a firstanswer category and a second set of the plurality of answers into asecond answer category. Each of the first set can correspond to at leastone of a third set of a plurality of confidence scores and each of thesecond set can correspond to at least one of a fourth set of theplurality of confidence scores. The plurality of confidence scores canrepresent confidence of answers to a query submitted to aquestion-answering system. The method can include classifying confidencescores of the third set into one of a plurality of confidence bucketsusing a first threshold and determining a fifth set of a plurality ofthresholds using the plurality of confidence scores. The method caninclude classifying unclassified confidence scores of the third set intoone of the plurality of confidence buckets using the fifth set of theplurality of thresholds.

The above summary is not intended to describe each illustratedembodiment or every implementation of the present disclosure.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The drawings included in the present application are incorporated into,and form part of, the specification. They illustrate embodiments of thepresent disclosure and, along with the description, serve to explain theprinciples of the disclosure. The drawings are only illustrative ofcertain embodiments and do not limit the disclosure.

FIG. 1 depicts a diagram of an example set of answer sequences,according to embodiments of the present disclosure.

FIG. 2 depicts a block diagram of an example computing environment foruse with a question-answering (QA) system, according to embodiments ofthe present disclosure.

FIG. 3 depicts a block diagram of an example QA system configured togenerate answers in response to one or more input queries, according toembodiments of the present disclosure.

FIG. 4 depicts a system architecture configured to manage answersgenerated by an example QA system, according to embodiments of thepresent disclosure.

FIG. 5 depicts a diagram of using answer management to generate one ormore answer sequences, according to embodiments of the presentdisclosure.

FIG. 6 depicts a flowchart diagram of a method of answer management in aQA environment, according to embodiments of the present disclosure.

FIG. 7 depicts a flowchart diagram of a method of answer relationshipmanagement in a QA environment, according to embodiments of the presentdisclosure.

FIG. 8 depicts a diagram of an example labeled answer sequence includingcharacteristic relationships, direct influence relationships, and answerrelationships, according to embodiments of the present disclosure.

FIG. 9 depicts a flowchart diagram of a method of evaluating an answersequence based on answer relationships, according to embodiments of thepresent disclosure.

FIG. 10 is a flowchart illustrating a method for managing answersequences, according to embodiments of the present disclosure.

FIG. 11 is a diagram illustrating an example system architecture formanaging answer sequences, according to embodiments of the presentdisclosure.

FIG. 12 depicts an example of answer sequence generation, according toembodiments of the present disclosure.

FIG. 13 depicts a conceptual diagram of a QA system configured toclassify answers sorted according to answer category, according toembodiments of the present disclosure.

FIG. 14 depicts a conceptual diagram of a QA system configured toclassify answers with buckets using multiple sets of thresholds,according to embodiments of the present disclosure.

FIG. 15 depicts a flow diagram illustrating example operations forassociating answer category confidence scores with confidence buckets,according to embodiments of the present disclosure.

FIG. 16 depicts a flow diagram illustrating example operations forassociating answer category confidence scores with confidence buckets,according to embodiments of the present disclosure.

FIG. 17 depicts a conceptual diagram illustrating a QA system configuredto distribute answers classified according to confidence buckets,according to embodiments of the present disclosure.

FIG. 18 is a flowchart illustrating a method for scoring answersequences, according to embodiments.

FIG. 19 is a high level flow-diagram of a method for scoring answersequences, according to embodiments.

While the invention is amenable to various modifications and alternativeforms, specifics thereof have been shown by way of example in thedrawings and will be described in detail. It should be understood,however, that the intention is not to limit the invention to theparticular embodiments described. On the contrary, the intention is tocover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the invention.

DETAILED DESCRIPTION

Aspects of the present disclosure relate to answer management in aquestion-answering (QA) environment, more particular aspects relate toestablishing an answer sequence from answers sorted according to asequence of answer categories. While the present disclosure is notnecessarily limited to such applications, various aspects of thedisclosure may be appreciated through a discussion of various examplesusing this context.

Embodiments of the present disclosure are directed towards a systemconfigured for answer management in a QA environment. In a QA system, agroup of answers can be generated in response to input queries (e.g.,questions). For example, the QA system can be configured to receive aninput query, analyze one or more data sources, and based on theanalysis, generate the group of answers.

In embodiments, answers can be data generated by a QA system in responseto an input query. Answers can be data in various forms including, butnot limited to, text, documents, images, video, and audio. Inembodiments, answers can be data that suggests an operation or action.For example, the QA system could receive a question asking how to treata particular medical condition. In response, the QA system couldgenerate a group of answers that collectively suggest a series or groupof actions for treating the particular medical condition. For example,the system could analyze a corpus of information and determine thatspecific medication could be used to treat the particular medicalcondition. In response, the system could generate an answer indicatingthat the specific medication should be taken. Described further herein,the QA system can generate answers based on natural language analysis ofa corpus of information.

In some instances, the QA system can be further configured to manageorganization of the group of answers. In embodiments, the organizedgroup of answers can be outputted to a user as a single, organized,complete answer (e.g., an answer sequence as described herein). In someembodiments, the system can be configured to render a visualization ofthe organized answer to present the answers to a user. Organizing theset of answers can assist a user in comprehension of the group ofanswers. In some embodiments, the group of answers can be organized invarious forms such as, but not limited to, images, charts, tables,dashboards, maps, and the like.

In some instances, answers from the set of answers can be scored with aconfidence value (e.g., a confidence score). The system can beconfigured to organize of the group of answers by generating an answerlist of the group of answers ordered according to the confidence valueof each answer. The answer list could then be presented, as an outputresponse, to satisfy the input query.

However, in some instances, the answer list could fail to satisfy theinput query. For example, the QA system could receive a question askinghow to treat a particular medical condition. In response, the QA systemcould generate the group of answers that suggest various actions. Thesystem could organize of the group of answers to form an answer listincluding the various treatments listed according to a confidence score.The answer list could be outputted to a user to attempt to satisfy theinput query. In some embodiments, the answers can be treatment answers,where treatment answers are answers that suggest various actions oroperations related to medical treatments.

However, the answer list can present answers such that it appears thatthe highest ranked answers in the list make up the suggested treatment.For example, a user, when seeing the answer list, could think that asingle answer (such as the one with the highest confidence score) is thesuggested treatment. However, a more desirable response could involve aplurality of treatments. For example, it could be that a combination oftwo answers, regardless of confidence score, presents a better answerthan a single answer. In an additional example, the user, when seeingthe answer list, could think that multiple answers (such as the top twoanswers) make up the suggested treatment, regardless of the category ortype of treatment suggested by the multiple answers. However, in someinstances, a more desirable response to an input query could involveapplying multiple categories or types of treatments. Additionally, amore desirable response could involve applying multiple answers in aparticular sequence.

For example, in the field of oncology, a more desirable response to aquestion of how to treat a specific cancer could generally involve twocategories of treatment answers. The categories could include aradiation treatment and a chemotherapy treatment. Additionally in someinstances, a more desirable response could include applying thecategories in a particular sequence. For example, an answer couldinclude first performing a radiation treatment and then a chemotherapytreatment. Additionally, in some instances, the categories could beapplied in an overlapping manner. For example, an answer could includefirst beginning a radiation treatment and then, prior to completing theradiation treatments, beginning a chemotherapy treatment.

Therefore, in embodiments, the system could be configured to manage thegroup of answers to organize answers according to a plurality of answercategories. In embodiments, answer categories are classifications thatcan be applied to the group of answers to assist in organization of theanswers.

For example, the answer categories could be used to classify the groupof answers according to type of action suggested by each answer. For agroup of answers generated in response to a question asking how totroubleshoot a computer, the answer categories could include hardwaretroubleshooting and software troubleshooting. Described further herein,the answer categories can be determined based on a subject matter ofdata (such as input queries and the generated answers) in the QAenvironment.

In embodiments, the system can be configured to sort the group ofanswers into a plurality of answer categories. For example, the systemcould sort a first set of the group of answers related to hardwaretroubleshooting into a first answer category which corresponds tohardware troubleshooting. The system could sort a second set of answersrelated to software troubleshooting into a second answer category, whichcorresponds to software troubleshooting.

In some embodiments, the answer categories can be ordered according to asequence. The sequence of answer categories can be referred to herein asa category sequence. For example, for an answer to an input queryrelated to cancer treatments, a category sequence could include orderedsteps of first applying radiation type treatments and then applyingchemotherapy type treatments. In an additional example, for an answer toan input query related to computer troubleshooting, a category sequencecould include ordered steps of first applying hardware troubleshootingand then software troubleshooting. Described further herein, thecategory sequences can be determined based on the subject matter of data(such as input queries and the generated answers) in the QA environment.

The system can be configured to establish, based on the one or morecategory sequences, one or more answer sequences. The one or more answersequences can be established from answers from one or more answercategories ordered according to the one or more category sequences. Forexample, a first set of answers could be sorted into a first answercategory and a second set of answers could be sorted into a secondanswer category. A category sequence could include the first answercategory followed by the second answer category. Thus, an answersequence could include a first answer from the first set of answersfollowed by a second answer from the second set of answers.

In some instances, a QA system could generate an answer sequence andpresent the answer sequence to a user without properly evaluating theinteractions between the answers that form the presented answersequence. This could lead to improper levels of confidence in the answersequence (e.g., confidence scores that are too high or too low). Forexample, in the field of oncology, a QA system could determine aconfidence score for a specific oncology treatment plan (answersequence) without considering how the specific treatments (answers) thatmake up the treatment plan are likely to interact. This could occur, forexample, where a confidence score for a treatment plan is generated as acomposite of the confidence scores of each specific treatment of thetreatment plan. In such a situation, unless the individual treatmentsare evaluated in view of their interactions with each other (e.g., wherethe individual treatments are not scored independently), the compositeconfidence score for the treatment plan could be inappropriate.

In some instances, a failure to take into account answer interactionscould lead to confidence scores that are too high. For example, in thefield of IT support, just because a particular computer troubleshootingplan (answer sequence) calls for using the debugger (first answer) withthe highest confidence score of all of the debuggers identified in theQA environment followed by using the network analyzer (second answer)with the highest confidence score of all of the network analyzersidentified in the QA environment does not mean that that particularcomputer troubleshooting plan is likely to be the best plan or even thatit is likely to be a good plan. There could be known (or at leastdiscoverable) negative interactions between the two answers (theparticular debugger and the particular network analyzer) that could beconsidered before recommending or presenting this particular plan to auser.

In some embodiments of the present disclosure, likely interactionsbetween answers of a particular answer sequence can be considered aspart of the ranking and/or scoring answer sequences. In someembodiments, this can involve generating an answer relationship in ananswer sequence. Specifically, this can occur by first identifying theanswer sequence, which can include at least a first answer and a secondanswer. Next, a corpus can be analyzed using the first answer and thesecond answer in order to identify a set of influence factors thatcorrespond to both answers. Based on this set of influence factors, theanswer relationship between the first answer and the second answer maybe generated.

In some embodiments, an answer sequence may include three or moreanswers. In such embodiments, answer relationships between each answerof the answer sequence and all of the remaining answers of the answersequence may be generated by identifying sets of influence factorsbetween each possible answer-answer pairing within the answer sequence.Each set of influence factors may be used to generate a separate answerrelationship. In some embodiments, the answer sequence may be evaluated,at least in part, based on the answer relationships between itsconstituent answers.

In some embodiments, a relationship score may be assigned to each answerrelationship based on its set of influence factors. Further, in someembodiments, the one or more relationship scores applied to the answerrelationships of a particular answer sequence, may impact the confidencescore of the answer sequence. Furthermore, in some embodiments,thresholds may be applied to relationship scores in order to determineif corresponding answer sequences are to be deemed improper, unusable,or otherwise contraindicated.

In some embodiments, identifying a set of influence factorscorresponding to both a first answer and a second answer of an answersequence may involve identifying a direct influence relationship betweenthe first answer and the second answer. Based on the direct influencerelationship, at least one influence factor of the set of influencefactors can be identified.

In some embodiments, identifying a set of influence factorscorresponding to both a first answer and a second answer of an answersequence may involve identifying a first characteristic relationshipbetween the first answer and a characteristic and a secondcharacteristic relationship between the second answer and thecharacteristic. The first characteristic relationship and the secondcharacteristic relationship may be compared in order to identify atleast one influence factor of the set of influence factors.

In recent years, the increased availability and access to large amountsof content via the Internet, social media, and other networks haveresulted in an increase in the need for organizing and managing thatcontent. As described herein, question-answering systems are one toolthat can be used to facilitate the ease with which users can find andaccess desired content. Aspects of the present disclosure, in certainembodiments, relate to the recognition that in certain situations,answers for questions submitted to the question answering system may bepart of a larger procedure or sequence of multiple answers (e.g., ananswer sequence), and that a single answer may not provide a completepicture of the desired content that the user is seeking. For instance,in the field of oncology, a user searching for the most effective cancertreatment may be overwhelmed by the number of treatment optionsavailable, and be unsure of which types of treatments work well with oneanother or in which order they should be applied. Accordingly, aspectsof the present disclosure, in certain embodiments, are directed towardanalyzing a corpus of data pertaining to a subject matter (e.g.,oncology) and determining an answer sequence for answers identified fromthe corpus. Further aspects of the present disclosure are directedtoward generating an answer sequence model for analyzing known answersequences and generating additional (e.g., undiscovered) answersequences. Aspects of the present disclosure may be associated withbenefits including content relevance, time saving, and efficiency ofcontent accessibility.

Embodiments of the present disclosure are directed towards a method formanaging category specific confidence scores in a QA environment. Inembodiments, the method can include sorting, based on a set of answercategories for a subject matter, a first set of a plurality of answersinto a first answer category and a second set of the plurality ofanswers into a second answer category.

In embodiments, each of the first set of the plurality of answerscorresponds to at least one of a set of a plurality of confidence scoresand each of the second set of the plurality of answers corresponds to atleast one of a fourth set of the plurality of confidence scores. Inembodiments, the plurality of confidence scores represent confidence ofanswers to an input query submitted to a QA system. In embodiments, themethod can include classifying confidence scores of the third set intoone of a plurality of confidence buckets using a first threshold. Themethod can include determining a fifth set of a plurality of thresholdsusing the plurality of confidence scores. The method can includeclassifying unclassified confidence scores of the third set into one ofthe plurality of confidence buckets using the fifth set of the pluralityof thresholds.

As described herein, a QA system can receive an input query and answersto that input query can be generated by the system. In embodiments, thesystem can be configured to generate corresponding answer confidencescores for one or more of the answers. In some instances, returning theanswers and confidence scores alone could overwhelm a user or lead tomisinterpretations of the quality of a returned answer, such as in ananswer list arrangement, as described herein.

Thus, in some instances, the system can be configured to sort theanswers into various answer categories, as described herein. Forexample, based on a set of answer categories for a subject matter, afirst set of a plurality of answers can be sorted into a first answercategory and a second set of the plurality of answers into a secondanswer category. In embodiments, each of the first set of the pluralityof answers can correspond to at least one of a third set of a pluralityof confidence scores. Similarly, in some embodiments, each of the secondset of the plurality of answers can correspond to at least one of afourth set of the plurality of confidence scores.

Additionally, the system can be configured to classify answers in eachof the answer categories into various confidence buckets. The answers ineach answer category can be classified based on a confidence scorecorresponding to each answer. In embodiments, confidence buckets aredivisions or classifications for answers based on a value of theanswer's confidence score.

For example, the system can be configured to classify the third set ofthe plurality of confidence scores to one or more confidence buckets.The system could be configured to classify the fourth set of theplurality of confidence scores to one or more confidence buckets.

In embodiments, confidence buckets can contain a group of answers and/orconfidence scores and can be associated with one or more thresholdvalues and a descriptive label. For example, answers that have aconfidence score above 95 on a scale of 0-100 could be classified into afirst bucket labeled “preferred answers”. Answers that have a confidencescore below 95 could be classified into a second bucket labeled “answersfor consideration”. Classifying answers into confidence buckets can bebeneficial, as the returned answers can be easier to display andinterpret. Confidence buckets can be referred to herein as “buckets”.

When using buckets, the QA system can determine which answers toassociate with which buckets by comparing the answer confidence scoresto bucket thresholds. In embodiments, static bucket thresholds can beused to allow answers to be presented according to accepted standards.For instance, an answer confidence above 95 on a scale of 0-100 couldattribute high confidence to the corresponding answer. Thus, in someinstances, confidence scores greater than 95 would be placed into a highconfidence bucket.

However, in some instances, using static bucket thresholds alone coulddisregard the relative value of a set of answers. For example, if allconfidence scores were greater than a static threshold of 95 on a scaleof 0-100, the confidence scores could end up classified into a singlebucket, such as the preferred answer bucket. A single bucket of answerscould only partially indicate or could not indicate relative confidenceof answers with respect to other answers.

Thus, in some instances, the system can be configured to use dynamicbucket thresholds based on the answer confidence scores to classify theconfidence scores. In embodiments, dynamic bucket thresholds are basedon answer confidence scores and the QA system can create bucketthresholds that can capture the relative confidence of the answers. Inaddition, using both static and dynamic bucket thresholds can allow thesystem to present answers in a manner that captures relative confidencewithin a framework of a standard of confidence.

As described herein, in certain situations, answers for questionssubmitted to the question answering system may be part of a largerprocedure or sequence of multiple answers (e.g., an answer sequence),and that a single answer may not provide a complete picture of thedesired content that the user is seeking. Often, the answers of theanswer sequence may be scored or ranked with confidence values or otherquantitative indications of the confidence or reliability of thatparticular answer.

Aspects of the present disclosure, in certain embodiments, relate to therecognition that it may be desirable to provide an overall compositescore (e.g., a sequence evaluation score) for the answer sequence as awhole based on the individual scores of the answers it includes.Furthermore, aspects of the present disclosure relate to the recognitionthat, depending on the subject matter that the answer sequence pertainsto, different methods of generating the sequence evaluation score may bedesirable (e.g., answer sequences pertaining to serious subject matterssuch as oncology, investment plans and the like may be evaluateddifferently than answer sequences related to entertainment, baking,etc.) Accordingly, aspects of the present disclosure are directed towarddetermining an evaluation rule for a particular answer sequence based onthe subject matter it relates to, as well as other conditions, andgenerating an overall composite score to indicate the reliability of theanswer sequence. Aspects of the present disclosure may be associatedwith benefits including content relevance, time saving, and efficiencyof content accessibility.

Referring now to FIG. 1 a diagram of an example table 100 showing answersequences can be seen, according to embodiments of the presentdisclosure. The table 100 can include a plurality of treatment answers110-128 organized according to various answer categories 102-108. Asseen in FIG. 1, the answer categories 102-108 are related to varioustypes of medical treatment categories. For example, answer category 102is related to chemotherapy, answer category 104 is related to surgery,answer category 106 is related to endocrine therapy, and answer category108 is related to radiation.

In embodiments, answer categories 102-108 can be referred to astreatment categories. In embodiments, treatment categories areclassifications, similar to answer categories, which are applied totreatment answers to assist in organization of treatment answers. Forexample, treatment answers 110 and 118 are related to chemotherapytreatments and thus are placed in a column underneath the treatmentcategory related to chemotherapy. Similarly, treatment answers 112, 120,and 124 are related to surgery treatments and thus are placed in acolumn underneath the treatment category related to surgery.

Answer categories 102-108 can be seen arranged in a row 109 in acategory sequence. The category sequence is a sequence of answercategories, as described herein. For example, in row 109 the categorysequence can include first answer category 102, then answer category104, then answer category 106 and then answer category 108. Inembodiments, a category sequence can be referred to as a treatmenttemplate. In embodiments, the treatment template can be the same orsubstantially similar to the category sequence. In some embodiments,treatment templates can be a specific category sequence that has beenidentified as acceptable or possible, either by an expert or by the QAsystem itself.

A set of answer sequences can be seen in rows 130-134. The set of answersequences are an ordered sequence of treatment answers (or answers),ordered based on a category sequence. Thus, in FIG. 1 a first answersequence can be seen in row 130 that includes treatment answer 110related to chemotherapy A, then treatment answer 112 related to surgeryA, then treatment answer 114 related to endocrine therapy A, thentreatment answer 116 related to radiation treatment A. The first answersequence suggests a treatment plan of the various treatment answers110-116 performed in order according to the category sequence 109. Inembodiments an answer sequence can be referred to as a treatment plan.In embodiments a treatment plan is an answer sequence generated fromtreatment answers ordered according to a treatment template, asdescribed herein.

In some embodiments, answer sequences can include answers from a portionof answer categories in a category sequence. For example, in row 132 and134, second and third answer sequences can be seen respectively. Thesecond answer sequence includes treatment answers 118, 120, 122 fromanswer categories 102, 104, and 108. The second answer sequence does notinclude a treatment answer from answer category 106.

Referring now to FIG. 2 a block diagram of an example computingenvironment 200 for use with a QA system can be seen, according toembodiments of the present disclosure. In some embodiments, thecomputing environment 200 can include one or more remote devices 202,212 and one or more host devices 222. Remote devices 202, 212 and hostdevice 222 can be distant from each other and communicate over a network250. In embodiments, the host device 222 can be a central hub from whichremote devices 202, 212 establish a communication connection. Inembodiments, the host device and remote devices can be configured invarious suitable relationships (e.g., in a peer-to-peer or otherrelationship).

In some embodiments, the network 250 can be implemented by suitablecommunications media (e.g., wide area network (WAN), local area network(LAN), Internet, and Intranet). In some embodiments, remote devices 202,212 and host devices 222 can be local to each other, and communicate viaappropriate local communication medium (e.g., local area network (LAN),hardwire, wireless link, Intranet). In some embodiments, the network 250can be implemented within a cloud computing environment, or using one ormore cloud computing services. Consistent with various embodiments, acloud computing environment can include a network-based, distributeddata processing system that provides one or more cloud computingservices. Further, a cloud computing environment can include multiplecomputers (e.g., hundreds or thousands of them or more), disposed withinone or more data centers and configured to share resources over thenetwork 250.

In some embodiments, host device 222 can include a QA system 230 havinga search application 234 and an answer module 232. The searchapplication 234 can be configured to search one or more databases orother computer systems for content that is related to an input query bya user at a remote device 202, 212.

In some embodiments, remote devices 202, 212 can enable users to submitinput queries (e.g., search requests or other user queries) to hostdevice 222 to retrieve search results. For example, the remote devices202, 212 can include a query module 210, 220 (e.g., in the form of a webbrowser or other suitable software module) and present a graphical userinterface or other interface (command line prompts, menu screens, etc.)to solicit queries from users for submission to one or more host devices222 and to display answers/results obtained from the host devices 222 inrelation to such user queries (e.g., answer sequences).

Consistent with various embodiments, host device 222 and remote devices202, 212 can be computer systems, and can each be equipped with adisplay or monitor. The computer systems can include at least oneprocessor 206, 216, 226; memories 208, 218, 228; internal or externalnetwork interface or communications devices 204, 214, 224 (e.g., modem,network interface cards); optional input devices (e.g., a keyboard,mouse, touchscreen, or other input device); and commercially availableor custom software (e.g., browser software, communications software,server software, natural language processing software, search engineand/or web crawling software, filter modules for filtering content basedupon predefined criteria). In some embodiments, the computer systems caninclude servers, desktops, laptops, and hand-held devices. In addition,the answer module 232 can include one or more modules or units toperform the various functions of embodiments as described below, and canbe implemented by a combination of software and/or hardware modules orunits.

Referring now to FIG. 3 a block diagram of a QA system can be seen,according to embodiments of the present disclosure. Aspects of FIG. 3are directed toward a system architecture 300, including a QA system 312to generate a group of answers (or groups of answer sequences) inresponse to an input query. In some embodiments, one or more users cansend requests for information to QA system 312 using a remote device(such as remote devices 202, 212 of FIG. 2). The remote device caninclude a client application 308 which can include one or more entitiesoperable to generate information that is dispatched to QA system 312 vianetwork 315. QA system 312 can be configured to perform methods andtechniques for responding to the requests sent by the client application308. In some embodiments, the information received at QA system 312 cancorrespond to input queries received from users, where the input queriescan be expressed in natural language, or images, or other forms.

An input query (similarly referred to herein as a question) can be oneor more words that form a search term or request for data, information,or knowledge. A question can be expressed in the form of one or morekeywords. Questions can include various selection criteria and searchterms. A question can be composed of complex linguistic features inaddition to keywords. However, a keyword-based search for answers canalso be possible. In some embodiments, using restricted syntax forquestions posed by users can be enabled. The use of restricted syntaxcan result in a variety of alternative expressions that assist users inbetter stating their needs. In some embodiments, questions can beimplied (rather than explicit) questions. Furthermore, in someembodiments, questions can be audio-type (e.g., spoken-word recordings,music, scientific sound recordings), video-type (e.g., a film, a silentmovie, a video of a person asking a detailed question), image-type(e.g., a picture, a photograph, a drawing), or other type that can bereceived and processed by the QA system.

In some embodiments, client application 308 can operate on a variety ofdevices. Such devices can include, but are not limited to, mobile andhand-held devices (e.g., laptops, mobile phones, personal or enterprisedigital assistants, and the like), personal computers, servers, or othercomputer systems that can access the services and functionality providedby QA system 312. In some embodiments, client application 308 caninclude one or more components, such as a mobile client 310. Mobileclient 310, acting as an agent of client application 308, can dispatchuser query requests to QA system 312.

Consistent with various embodiments, client application 308 can alsoinclude a search application 302, either as part of mobile client 310 orseparately, that can perform several functions, including some or all ofthe above functions of mobile client 310 listed above. For example, insome embodiments, search application 302 can dispatch requests forinformation to QA system 312. In some embodiments, search application302 can be a client application to QA system 312. Search application 302can send requests for answers to QA system 312. Search application 302can be installed on a personal computer, a server, or other computersystem.

In some embodiments, search application 302 can include a searchgraphical user interface (GUI) 304 and session manager 306. In suchsituations, users can be able to enter questions in search GUI 304. Insome embodiments, search GUI 304 can be a search box or other GUIcomponent, the content of which can represent a question to be submittedto QA system 312. Users can authenticate to QA system 312 via sessionmanager 306. In some embodiments, session manager 306 can keep track ofuser activity across sessions of interaction with the QA system 312.Session manager 306 can also keep track of what questions are submittedwithin the lifecycle of a session of a user. For example, sessionmanager 306 can retain a succession of questions posed by a user duringa session. In some embodiments, answers produced by QA system 312 inresponse to questions posed throughout the course of a user session canalso be retained. Information for sessions managed by session manager306 can be shared between various computer systems and devices.

In some embodiments, client application 308 and QA system 312 can becommunicatively coupled through network 315, e.g., the Internet,intranet, or other public or private computer network. In someembodiments, QA system 312 and client application 308 can communicate byusing Hypertext Transfer Protocol (HTTP) or Representational StateTransfer (REST) calls. In some embodiments, QA system 312 can reside ona server node. Client application 308 can establish server-clientcommunication with QA system 312 or vice versa. In some embodiments, thenetwork 315 can be implemented within a cloud computing environment, orusing one or more cloud computing services.

Consistent with various embodiments, QA system 312 can respond to arequest for information sent by client applications 308 (e.g., questionposed by a user). QA system 312 can generate a group of answers inresponse to the request. In some embodiments, QA system 312 can includea question analyzer 314, data sources 324, and answer generator 328.Question analyzer 314 can be a computer module that analyzes thereceived questions. Question analyzer 314 can perform various methodsand techniques for analyzing the questions (syntactic analysis, semanticanalysis, image recognition analysis, etc.). In some embodiments,question analyzer 314 can parse received questions. Question analyzer314 can include various modules to perform analyses of receivedquestions. For example, computer modules that question analyzer 314 canencompass include, but are not limited to, a tokenizer 316,part-of-speech (POS) tagger 318, semantic relationship identifier 320,and syntactic relationship identifier 322.

In some embodiments, tokenizer 316 can be a computer module thatperforms lexical analysis. Tokenizer 316 can convert a sequence ofcharacters into a sequence of tokens. A token can be a string ofcharacters typed by a user and categorized as a meaningful symbol.Further, in some embodiments, tokenizer 316 can identify word boundariesin an input query and break the question or text into its componentparts such as words, multiword tokens, numbers, and punctuation marks.In some embodiments, tokenizer 316 can receive a string of characters,identify the lexemes in the string, and categorize them into tokens.

Consistent with various embodiments, POS tagger 318 can be a computermodule that marks up a word in a text to correspond to a particular partof speech. POS tagger 318 can read a question or other text in naturallanguage and assign a part of speech to each word or other token. POStagger 318 can determine the part of speech to which a word correspondsbased on the definition of the word and the context of the word. Thecontext of a word can be based on its relationship with adjacent andrelated words in a phrase, sentence, question, or paragraph. In someembodiments, the context of a word can be dependent on one or morepreviously posed questions. Examples of parts of speech that can beassigned to words include, but are not limited to, nouns, verbs,adjectives, adverbs, and the like. Examples of other part of speechcategories that POS tagger 318 can assign include, but are not limitedto, comparative or superlative adverbs, wh-adverbs, conjunctions,determiners, negative particles, possessive markers, prepositions,wh-pronouns, and the like. In some embodiments, POS tagger 318 can tagor otherwise annotate tokens of a question with part of speechcategories. In some embodiments, POS tagger 318 can tag tokens or wordsof a question to be parsed by QA system 312.

In some embodiments, semantic relationship identifier 320 can be acomputer module that can identify semantic relationships of recognizedentities (e.g., words, phrases) in questions posed by users. In someembodiments, semantic relationship identifier 320 can determinefunctional dependencies between entities and other semanticrelationships.

Consistent with various embodiments, syntactic relationship identifier322 can be a computer module that can identify syntactic relationshipsin a question composed of tokens posed by users to QA system 312.Syntactic relationship identifier 322 can determine the grammaticalstructure of sentences, for example, which groups of words areassociated as “phrases” and which word is the subject or object of averb. Syntactic relationship identifier 322 can conform to formalgrammar.

In some embodiments, question analyzer 314 can be a computer module thatcan parse a received user query and generate a corresponding datastructure of the user query. For example, in response to receiving aquestion at QA system 312, question analyzer 314 can output the parsedquestion as a data structure. In some embodiments, the parsed questioncan be represented in the form of a parse tree or other graph structure.To generate the parsed question, question analyzer 314 can triggercomputer modules 316-322. Additionally, in some embodiments, questionanalyzer 314 can use external computer systems for dedicated tasks thatare part of the question parsing process.

In some embodiments, the output of question analyzer 314 can be used byQA system 312 to perform a search of a set of (i.e., one or more)corpora to retrieve information to answer a question posed by a user. Asused herein, a corpus can refer to one or more data sources. In someembodiments, data sources 324 can include databases, informationcorpora, data models, and document repositories. In some embodiments,the data source 324 can include an information corpus 326. Theinformation corpus 326 can enable data storage and retrieval. In someembodiments, the information corpus 326 can be a storage mechanism thathouses a standardized, consistent, clean and integrated form of data.The data can be sourced from various operational systems. Data stored inthe information corpus 326 can be structured in a way to specificallyaddress reporting and analytic requirements. In some embodiments, theinformation corpus can be a relational database. In some exampleembodiments, data sources 324 can include one or more documentrepositories.

In some embodiments, answer generator 328 can be a computer module thatgenerates the group of answers in response to posed questions. Examplesof answers generated by answer generator 328 can include, but are notlimited to, natural language sentences, reports, charts, or otheranalytic representation, raw data, web pages, and the like. In someembodiments, answers can be of audio type, image type, or other suitablemedium type.

In some embodiments, answer generator 328 can include query processor330, answer management processor 332, and feedback handler 334. Wheninformation in the data source 324 matching a parsed question islocated, a technical query associated with the pattern can be executedby query processor 330. Based on data retrieved by a technical queryexecuted by query processor 330, answer management processor 332 can beconfigured to organize the retrieved answers. In embodiments, the answermanagement processor 332 can be a visualization processor configured torender a visualization of the organized answers. In embodiments, therendered visualization of the answers can represent the answer to theinput query. In some embodiments, answer management processor 332 canorganize the answers according to various forms including, but notlimited to, images, charts, tables, dashboards, maps, and the like.

Described further herein, the answer management processor 332 can beconfigured to implement embodiments of the present disclosure. Forexample, the answer management processor 332 can be configured to sort,based on a set of answer categories, a first set of answers into a firstanswer category and a second set of answers into a second answercategory. The answer categories can be the same or substantially similaras described herein.

The answer management processor 332 can be configured to determine,using the subject matter, a category sequence including the first answercategory and the second answer category. The answer management processor332 can be configured to establish, based on the category sequence, afirst answer sequence established from a portion of the first set ofanswers from the first answer category and a portion of the second setof answers from the second answer category.

In some embodiments, feedback handler 334 can be a computer module thatprocesses feedback from users on answers generated by answer generator328. In some embodiments, users can be engaged in dialog with the QAsystem 312 to evaluate the relevance of received answers. For example,the answer generator 328 could produce the group of answerscorresponding to a question submitted by a user. The user could rankeach answer according to its relevance to the question. In someembodiments, the feedback of users on generated answers can be used forfuture question answering sessions.

The various components of the exemplary QA system described above can beused to implement various aspects of the present disclosure. Forexample, the client application 308 could be used to receive an inputquery from a user. The question analyzer 314 could, in some embodiments,be used to analyze input queries and to generate the group of answersbased on the input query. The answer generator 328 could, in someembodiments, be used to render visualization of the group of answers togenerate an answer sequence for presentation to the user.

Referring now to FIG. 4, a block diagram of a system architecture 400for answer management in a question-answering (QA) environment can beseen, according to embodiments of the present disclosure. Inembodiments, the system architecture 400 can represent an examplearchitecture for executing embodiments of the present disclosure. Forexample, in some instances, the system architecture 400 could be anexample representation of the answer management processor 332 (FIG. 3).

In embodiments, the system architecture 400 can include a subject matterprocessor 402, an answer categorizer 408, and an answer sorter 414.

The subject matter processor 402 can be a computer module configured todetermine a subject matter for data in the QA environment. As describedherein, data in the QA environment can include one or more input queriesand/or the group of answers generated in response to the input queries.In embodiments, the subject matter can be contextual information for thedata in the QA environment. The subject matter can be used to organizethe group of answers, as described herein. For example, describedfurther herein, the subject matter can be used to determine one or moreanswer categories for the group of answers. In some examples, thesubject matter can be used to determine one or more category sequences.For example, if the subject matter is oncology then the sequences mayinclude chemotherapy treatments and radiation treatments, but analternative category of computer troubleshooting might be left outbecause it is irrelevant to oncology.

In embodiments, the subject matter processor 402 can determine thesubject matter by receiving a subject matter selection from a user. Forexample, the user could select computer troubleshooting as the subjectmatter of data in the QA environment. A system could then actaccordingly in determining answer categories and/or category sequences,described further herein.

In some embodiments, the subject matter processor 402 can be configuredto determine the subject matter based on natural language analysis ofdata in the QA environment.

In embodiments, the subject matter processor 402 can include a naturallanguage processor 404. The natural language processor 404 can beconfigured to perform various methods and techniques for naturallanguage analysis of data in the QA environment. For example, thenatural language processor 404 can be configured to perform syntacticanalysis, semantic analysis, image recognition analysis, conceptmatching and other suitable methods and techniques.

In embodiments, the subject matter can be determined by concept matchingtechniques. Concept matching techniques can include, but is not limitedto, semantic similarity, syntactic analysis, and ontological matching.For example, in embodiments, the natural language processor could beconfigured to parse data in the QA environment to determine semanticfeatures (e.g. repeated words, keywords, etc.) and/or syntactic features(e.g. location of semantic features in headings, title, etc.) in thedata. Ontological matching could be used to map semantic and/orsyntactic features to a particular concept. The concept can then be usedto determine the subject matter for the data.

For example, in some embodiments, the natural language processor 404 canbe configured to parse the group of answers generated in response to theinput query. Natural language processor 404 could identify, in the groupof answers, repeated words corresponding to a particular type of cancer.Additionally, the natural language processor 404 could identify thelocation of the repeated words in headings and titles, which canindicate the relative importance of the repeated words. Based on thesemantic and syntactic features the natural language processor 404 couldmap the group of answers to a particular concept, such as oncology. Inembodiments, the subject matter processor 402 could be configured toselect the concept as the subject matter.

The answer categorizer 408 can be configured to determine a set ofanswer categories for the group of answers. As described herein, theanswer categories are classifications that can be applied to the groupof answers to assist in organization of the answers. For example, thegroup of answers generated in response to a question about how totroubleshoot a computer could include answers related to troubleshootinghardware and troubleshooting software. A first set of answerscorresponding to hardware troubleshooting could be sorted into a firstanswer category corresponding to hardware troubleshooting. A second setof answers corresponding to software troubleshooting could be sortedinto a second answer category corresponding to software troubleshooting.

Additionally, the answer categorizer 408 can be configured to determinea category sequence for the answer categories. The answer categorizercan include an answer category processor 410 and a category sequenceprocessor 412.

The answer category processor 410 can be configured to determine one ormore answer categories for the group of answers. In embodiments, theanswer categories can be determined based on the subject matter of datain the QA environment. For example, a subject matter related to oncologycould have different answer categories than a subject matter related tocomputer troubleshooting. In some embodiments, answer categories can beshared between subject matter. In embodiments, the answer categoryprocessor 410 can use the subject matter determination from the subjectmatter processor 402 to determine the one or more answer categories.

In embodiments, the answer category processor 410 can determine one ormore answer categories by accessing a repository of predefined answercategories. In embodiments, the repository of predefined answercategories can be stored in a database 413. In embodiments, the database413 can include one or more answer categories that correspond to varioussubject matter. For example, a set of answer categories includingradiation, chemotherapy, endocrine therapy, and surgery could correspondto the subject matter of oncology. Thus, when the subject matter isoncology, the answer category processor 410 could access the set ofanswer categories corresponding to oncology. Additionally, a set ofanswer categories including hardware troubleshooting and softwaretroubleshooting could correspond to the subject matter of IT support. Inembodiments, various suitable answer categories can also be selected forvarious subject matter.

In some embodiments, the answer category processor 410 can determine theanswer categories based on natural language analysis of data in the QAenvironment. For example, in embodiments, the answer category processor410 could be configured to analyze the input query, using a naturallanguage processing technique. Based on the analysis, the answercategory processor 410 could determine the answer categories.

In some embodiments, the answer category processor 410 could beconfigured to analyze the group of answers, using a natural languageprocessing technique. Based on the analysis, the answer categoryprocessor 410 could determine the answer categories.

The category sequence processor 412 can be configured to determine oneor more category sequences. In embodiments, the category sequenceprocessor 412 can be configured to determine the one or more categorysequences based on the subject matter. In embodiments, the categorysequence processor 412 can determine one or more category sequences byaccessing a repository of predefined category sequences. In embodiments,the repository of predefined category sequences can be stored in adatabase 413. In embodiments, the database 413 can include one or morecategory sequences that correspond to various subject matters. Forexample, a category sequence of first surgery, then radiation, thenchemotherapy, and then endocrine therapy could correspond to the subjectmatter of oncology. In embodiments, various category sequences can beselected for various subject matters. In some embodiments, a categorysequence processor may be able to weed out/not include categorysequences that are not relevant or are impractical.

The answer sorter 414 can be configured to sort the group of answersinto the various answer categories. The answer sorter 414 can sort thegroup of answers by classifying answers as related to one or more of theanswer categories. For example, the answer sorter 414 could sort a firstset of answers into a first answer category by classifying the first setof answers as related to the first answer category.

In embodiments, the answer sorter 414 can use natural language analysisto sort the group of answers. For example, in embodiments, the answersorter 414 can parse the group of answers to identify semantic featureswhich correspond to one or more of the answer categories. The answersorter 414 could then sort answers of the group of answers into answercategories that correspond to the identified semantic features.

In some embodiments, the answer sorter can sort the group of answersusing concept matching techniques, as described herein.

The answer sorter can include an answer sequencer 416. The answersequencer 416 can be configured to generate one or more answersequences. In embodiments, the answer sequencer 416 can generate the oneor more answer sequences based on the group of answers and the one ormore category sequences. For example, the answer sequencer can assemblean answer sequence including the group of answers from each answercategory included within a given category sequence, the group of answersordered based on a category sequence.

In an additional example, the answer sorter 414 could sort a first setof answers into a first answer category and a second set of answers intoa second answer category. From the category sequence processor 412, acategory sequence could include the first answer category followed bythe second answer category. The answer sequencer 416 could generate oneor more answer sequences from the first and second sets of answers. Forexample, an answer sequence could include a first answer from the firstset of answers followed by a second answer from the second set ofanswers. In embodiments, the answer sequencer could generate variouspossible combinations of answers in the first and second set of answersto generate the one or more answer sequences. In embodiments, the one ormore answer sequences can then be presented as an answer to an inputquery.

Referring now to FIG. 5, a diagram 500 of answer management can be seenaccording to embodiments of the present disclosure. The diagram depictsa system including a subject matter processor 506, an answer categoryprocessor 508, a category sequencer 512, an answer sorter 514, and ananswer sequencer 516.

Data in the QA environment, such as an input query 502 and the group ofanswers 504 generated in response to the input query 502, can beinputted to the subject matter processor 506. The subject matterprocessor 506 can be the same or substantially similar to the subjectmatter processor 402 (FIG. 4) as described herein. The subject matterprocessor 506 can be configured to determine a subject matter for datain the QA environment, and the subject matter can be used, as describedherein, to determine answer categories and category sequences for the QAsystem.

The answer category processor 508 can be configured to determine answercategories for the QA system. The answer category processor 508 can bethe same or substantially similar to answer category processor 410 (FIG.4). The answer category processor can determine a set of answercategories 510A-510C by accessing a database of answer categoriescorresponding to the subject matter.

Category sequencer 512 can be configured to determine a categorysequence of the answer categories 510A-510C. For example, categorysequencer 512 could determine a category sequence of the first answercategory 510A, then the third answer category 510C, and then the secondanswer category 510B. In embodiments, the category sequencer 512 candetermine the category sequence by accessing a database of categorysequences corresponding to the subject matter.

Answer sorter 514 can be configured to sort the group of answers 504into the answer categories 510A-510C. Answer sorter 514 can be the sameor substantially similar as the answer sorter 414 (FIG. 4). As seen inFIG. 5, answer sorter 514 can be configured to sort the group of answers504 into the set of answer categories 510A-510C to form a set of sortedanswers 515. For example, answer A and answer E are sorted into answercategory 510A. Answer B and answer C are sorted into answer category510C, and answer D is sorted into category 510B.

The answer sequencer 516 can be configured to generate one or moreanswer sequences 517 from the set of sorted answers 515. The answersequencer 517 can be the same or substantially similar as the answersequencer 416 (FIG. 4). The answer sequencer can be configured togenerate one or more answer sequences 517 by selecting an answer fromone or more answer categories in order according to the categorysequence. For example, the one or more answer sequences 517 couldinclude an answer sequence of answer A, then answer B, and then answerD. As seen in FIG. 5, answer sequencer 516 can form the one or moreanswer sequences 517 from various combinations of the sorted answers 515in order according to the category sequence. The one or more answersequences can be presented to a user to satisfy the input query 502.

Referring now to FIG. 6, a flowchart diagram of a method 600 of answermanagement in a question-answering (QA) environment can be seenaccording to embodiments of the present disclosure. In operation 602, aninput query can be received. The input query can be a request for datato a QA system from a user. The input query can be the same orsubstantially similar as described herein. In operation 604, a group ofanswers can be generated. The group of answers can be generated by ananswer generator in the QA system by retrieving answers from datasources, such as databases and/or information corpora.

In operation 606, a subject matter can be determined. The subject mattercan be the same or substantially similar as described herein. Thesubject matter can be contextual information related to data in the QAenvironment. For example, in embodiments, the subject matter could bethe topic of the input query. In some examples, the subject matter couldbe the topic of the group of answers generated in response to the inputquery.

In operation 608, a set of answer categories can be determined. The setof answer categories can be the same or substantially similar asdescribed herein. The answer categories can be classifications for thegroup of answers to assist in organization of the answers. Inembodiments, an answer category can be a high level description of anaction suggested by an answer, as described herein.

In operation 610, the group of answers can be sorted into the set ofanswer categories. The group of answers can be sorted by classifyinganswers as related to one or more of the answer categories. For example,a first set of answers could be sorted into a first answer category byclassifying the first set of answers as related to the first answercategory. In embodiments, the answers can be sorted into the answercategories using natural language analysis as described herein.

In operation 612, a set of category sequences can be determined. Thecategory sequences can be the same or substantially similar as describedherein. Described herein, the category sequence can be various sequencesof answer categories. As described herein, the category sequences can bedetermined based on the subject matter. In embodiments, the set ofcategory sequences can be accessed from a database by a QA system. Forexample, one or more category sequences could be predetermined andstored for access when the QA system is tasked with a subject mattercorresponding to the set of category sequences.

In operation 614, an answer sequence can be established. The answersequence can be the same or substantially similar as described herein.As described herein, the answer sequence can be formed by selecting thegroup of answers from one or more answer categories in order accordingto the category sequence.

Referring now to FIG. 7, a flowchart diagram of a method 700 of answerrelationship management in a QA environment can be seen according toembodiments of the present disclosure. In operation 702, an answersequence can be identified. The answer sequence can include any numberof answers. In some embodiments, the answer sequence can be generatedusing some or all of the operations of method 600 as shown in FIG. 6. Inoperation 704, a corpus can be analyzed using the answers of the answersequence. In some embodiments, this can take the form of a keywordsearch with the answers acting as keywords. Further, in someembodiments, the analysis can include parsing the corpus based on theanswers.

In operations 706-714, influence factors can be identified throughdirect influence relationship evaluations (per operations 706 and 708)and/or through characteristic relationship evaluations (per operations710, 712, and 714). In some embodiments, influence factors may beidentified based on sentiment factors (which are described elsewhereherein) associated with two or more answers. Further, in someembodiments, influence factors may be the same or substantially similarto influence components (which are also described elsewhere herein). Asdescribed herein, an influence factor can be an interaction or resultthat is likely to occur if two answers of an answer-answer pair of ananswer sequence are both used as provided for in that particular answersequence. Further, an influence factor can be a description or anevaluation (in terms of positive or negative, likely or unlikely, etc.)of an effect that one answer is known to have an another answer (onedirection influence) or that two answers are known to have on each other(two direction influence). Further, an influence factor can be a measureof or information about the compatibility of two answers of an answersequence that is inferred based on the interactions between each of thetwo answers and one or more common (e.g., shared) concepts. As anexample in the field of baking, consider a scenario wherein an answersequence includes a first answer of “add ingredient A” and a secondanswer of “stir immediately”. In this scenario several differentinfluence factors are possible. For example, if ingredient A gets badlyclumpy if it is stirred immediately, then influence factors of “likelyto causing clumping of ingredient A” or “second answer likely to causenegative influence on first answer” are possible.

In operation 706, direct influence relationships within answer-answerpairs can be identified based on the analysis of the corpus. Asdescribed herein, a direct influence relationship can be an explicit,immediate relationship between the answers of the particularanswer-answer pair. Further, a direct influence relationship can also bea first-degree connection between the answers of the answer-answer pairas discovered based on the analysis of the corpus. For example, in thefield of oncology, in an answer sequence including a first answer of“treat patient with chemotherapy A for four weeks” and a second answerof “treat patient with endocrine therapy Y”, there could be a directinfluence relationship between the first and second answers that couldbe discovered in a corpus (e.g., a medical journal) that includes apassage stating that “[a] patient should not be treated with endocrinetherapy Y if the patient has received or will receive more than one weekof chemotherapy A.” In operation 708, influence factors can beidentified based on the direct influence relationships identified inoperation 706. In this oncology example, a strongly negative influencefactor could be identified as corresponding to the first answer and thesecond answer based on the medical journal passage.

In operation 710, characteristic relationships between answers andcharacteristics can be identified based on the analysis of the corpus.As described herein, a characteristic can refer to an element, feature,or trait. Further as described herein, a characteristic relationship canrefer to a relationship between a particular answer of an answersequence and a particular characteristic. In some embodiments, acharacteristic relationship can include or be labeled with attributesthat describe, are evidence of, and/or quantify the nature of therelationship between the answer and the characteristic. For example, inthe field of IT support, an answer of “install new CPU” could havecharacteristic relationships with characteristics of “expensive” and“easy to perform” (e.g., where there is a first relationship between astep of installing a new CPU and a characteristic of being expensive andwhere there is a second relationship between the step of installing anew CPU and the characteristic of being easy to perform). In thisexample, the characteristic relationship between “install new CPU” and“expensive” could include the attribute of “approximately $700” (e.g.,where having a cost of approximately $700 is evidence of why installinga new CPU has a relationship with the characteristic of expensive) andthe characteristic relationship between “install new CPU” and “easy toperform” could include a negative correlation (e.g., where installing anew CPU is considered not easy to perform).

In operation 712, comparisons can be made between characteristicrelationships having common (e.g. shared) characteristics and different(e.g., non-shared) answers within an answer sequence. In operation 714,based on the comparisons of these characteristic relationships,influence factors can be identified as corresponding to the answers ofthese characteristic relationships. The comparison of characteristicrelationships is described in reference to FIG. 8.

In operation 716, influence factors identified in operations 706-714 canbe grouped into sets of influence factors based on the answer-answerpair to which each influence factor belongs. For example, in an answersequence including answers E, F, G, and H, there can be, in someembodiments, up to six different answer pairs (E-F, E-G, E-H, F-G, F-H,and G-H) and, therefore, up to six different sets of influence factorsinto which a given influence factor could be grouped. In operation 718,answer relationships are generated for each possible answer-answer pairbased on the set of influence factors corresponding to both answers ofthat answer-answer pair. Each answer relationship can represent acomposite of a particular set of influence factors. In some embodiments,answer relationships can be measures or indicators as to how answers arelikely to interact or influence each other (or influence the answersequence as a whole) if the answer sequence is used. Further, in someembodiments, for answer-answer pairs having no shared influence factors,there can be deemed to be no answer relationship between those answersforming the pair or there can be deemed to be a null or neutral answerrelationship. For instance, to continue the EFGH example above, if thereare no influence factors corresponding to the E-F pair then therelationship between answer E and answer F may be deemed a neutralanswer relationship. In operation 720, the identified answer sequencecan be evaluated based on the answer relationships.

To aid understanding, a simplified version of method 700 is performed inan example scenario. In this example, a question of “What steps should Itake to get a beautiful lawn on my property in Arizona?” is provided bya homeowner to a QA system. The QA system identifies several answersequences (per operation 702). One of the answer sequences includes afirst answer of “plant grass variety X in the spring” and a secondanswer of “add fertilizer Y to the lawn in the summer”. In this example,both answers are included in the answer sequence at least in partbecause the QA system determines that they both work well in hot, dryclimates. A corpus of lawn and gardening magazines is analyzed by the QAsystem using the two answers (per operation 704). In the analysis, apassage is discovered that states that “[f]ertilizer Y has been shown towork poorly on some lawns having grass variety X.” Based on thispassage, a direct influence relationship between the answers isidentified (per operation 706). Based on the direct influencerelationship a negative influence factor corresponding to both answersis identified (per operation 708). Also based on the analysis of thecorpus, characteristic relationships are identified between each answerand a characteristic of “tolerates hot climates” (per operation 710).Because these characteristic relationships have this sharedcharacteristic, they are compared (per operation 712). Based on thecomparison, a positive characteristic-based influence factor isidentified as corresponding to both answers (per operation 714). Thedirect influence factor and the characteristic-based influence factorare grouped together to form the set of influence factors correspondingto both answers (per operation 716). Based on the set of influencefactors (in this instance, the two influence factors), an answerrelationship is generated between the two answers (per operation 718).In this example, the negative direct influence factor and the positivecharacteristic-based influence factor are weighed against each other,but overall the negative influence factor is weighted more heavily(e.g., where the negative influence factor is determined to be moreinfluential) and the resulting answer relationship is negative. Based onthe answer relationship, the answer sequence is evaluated (per operation720). In this instance, because of the negative answer relationship, theconfidence score of the answer sequence is decreased and, as a result,this particular answer sequence is presented to the homeowner with alower ranking (relative to the other answer sequences) than would havebeen the case had the answer relationship not been considered.

Referring now to FIG. 8, a diagram of an example labeled answer sequence800 including characteristic relationships, direct influencerelationships, and answer relationships can be seen, according toembodiments of the present disclosure. As shown, example answer sequence800 includes answer A 801, answer B 802, and answer C 803. In someembodiments, this means that answer sequence 800 could include amultitude of different orderings or combinations of these three answers(answer A followed by answer B followed by answer C, answer A and answerC occurring at substantially the same time followed by answer B, etc.).In some embodiments, the exact ordering of the answers may or may notmatter for the purpose of establishing answer relationships (forexample, in some embodiments, answer sequence ABC could be treated thesame as answer sequence BCA).

As shown, there is only one direct influence factor of interest inevaluating answer sequence 800. Specifically, there is a directinfluence factor A/B 811 corresponding to both answer A 801 and answer B802. This direct influence factor A/B 811 can be based on a directinfluence relationship between answer A 801 and answer B 802. Also shownare four characteristics (a, b, c, and d) 807-810 and six characteristicrelationships (A/a, A/b, B/b, B/c, B/d, and C/d) 814-819. Two pairs ofcharacteristic relationships (A/b and B/b, B/d and C/d), 815 and 816,and 818 and 819, have common characteristics (b and d, respectively),808 and 810, and different answers. By comparing these pairs ofcharacteristic relationships, two characteristic-based influence factorscan be identified, namely, characteristic b-based influence factor 812corresponding to both answer A 801 and answer B 802 and characteristicd-based influence factor 813 corresponding to both answer B 802 andanswer C 803.

Further, as shown, answer relationships can be generated based on thesets of influence factors. Specifically, a first set of influencefactors (including the characteristic b-based influence factor 812 anddirect influence factor A/B 811) can be used to generate an answerrelationship A/B 804 between answer A 801 and answer B 802. Similarly, asecond set of influence factors (including characteristic d-basedinfluence factor 813) can be used to generate an answer relationship B/C805 between answer B 802 and answer C 803. In addition, because thereare no influence factors corresponding to both answer A 801 and answer C803, answer relationship A/C 806 can, in some embodiments, be deemednon-existent or neutral. Once each of the answer relationships 804-806have been generated, they can be used to evaluate answer sequence 800.

Referring now to FIG. 9, a flowchart diagram of a method 900 ofevaluating an answer sequence based on answer relationships can be seen,according to embodiments of the present disclosure. In operation 902, ananswer sequence is identified. In operation 904, answer relationships ofthe answer sequence can be identified. In some embodiments, operation904 may involve performing some or all of the operations of method 700shown in FIG. 7. In operation 906, a relationship score can be assignedto each answer relationship of the answer sequence. As described herein,a relationship score can indicate a measure of the impact that twoanswers are likely to have on each other or how well they are likely tointeract in a given answer sequence. Relationship scores can be positiveor negative (e.g., favorable or not favorable). In some embodiments,relationship scores can be based on influence factors. Further, in someembodiments, answer relationship scoring rules may be used to determinerelationship scores.

In decision block 908, a determination can be made as to whether thereare any relationship scores below a relationship contraindicationthreshold. As described herein, a relationship contraindicationthreshold can refer to a minimal acceptable level for a relationshipscore (e.g., the most negative that a relationship score can be whilestill being acceptable). If a given relationship score is below thisthreshold, then the answer sequence with which the given relationshipscore is associated may be contraindicated. As described herein, ananswer sequence may be considered contraindicated when it is deemedunusable or improper as a result of a negative evaluation of an answerrelationship for answers of that particular answer sequence. In someembodiments, employing such a threshold can help to ensure that astrongly negative relationship between two answers of an answer sequencecan prevent the answer from being recommended to a user. In embodiments,relationship contraindication thresholds can be more tolerant or lesstolerant of negative relationship scores. A less tolerant threshold canbe applied, for example, in situations where it is more important to besure that negative interactions between answers of a particular answersequence are limited if that answer sequence is to be presented to auser (e.g., in a medical treatment setting).

If in operation 908, at least one relationship score is below thethreshold, then, in operation 910, the entire answer sequence may beidentified as contraindicated. In some embodiments, thiscontraindication identification may mean that the answer sequence is noteven presented to the user as a possible answer sequence; or, in otherembodiments, the answer sequence may only be presented along with awarning label and a description of the reason for the contraindication.As an example, consider a generic answer sequence of JKLM. If an answerrelationship between L and M has a relationship score below a threshold,then the answer sequence JKLM may be identified as contraindicated eventhough all of the remaining answer relationships (between J and L,between K and M, etc.) are all associated with relationship scores abovethe threshold.

If in operation 908, all of the relationship scores are above therelationship contraindication threshold, then, per operation 912, aconfidence score can be assigned to the answer sequence. The confidencescore can be based in part on the relationship scores associated withthe answer sequence. In some embodiments where an original confidencescore has been assigned to the answer sequence prior to the answerrelationship evaluation, a revised confidence score can be assigned. Therevised confidence score can be based on both the original confidencescore and the relationship scores.

FIG. 10 is a flowchart illustrating a method 1000 for managing answersequences, consistent with embodiments of the present disclosure.Aspects of FIG. 10 are directed toward determining a first answersequence using ordering data for a first set of answers. The method 1000may begin at block 1002 and end at block 1012. Consistent with variousembodiments, the method 1000 may include a parsing block 1004, adetecting block 1006, an identifying block 1008, and a determining block1010.

Consistent with various embodiments, at block 1004 the method 1000 mayinclude parsing, by a natural language processing technique, a corpus ofdata for a subject matter. The subject matter may include content ordata related to particular topic, theme, or concept. The naturallanguage processing technique may be configured to parse syntactic andsemantic data of the corpus of data. In certain embodiments, the corpusof data for the subject matter may be a database including one or moretypes of content related to a particular topic or subject. The types ofcontent may include, for instance, research results, practice trialresults, journal articles, historical data, or the like. For example, incertain embodiments, the database may include medical research trials,journal articles and other sorts of content relating to a subject matterof oncology treatment. As an additional example, the database mayinclude content related to one or more other subjects, such asgardening, computer technical support, or beekeeping. Other subjectmatters are also possible. In certain embodiments, the subject mattercontent on the database may be organized, classified, and tagged. Forinstance, the subject matter content on the database may be organized orstructured by linking concepts and subtopics together using an ontologyframework. In certain embodiments, the corpus of data may correspond toinformation corpus 326 of FIG. 3.

As described herein, at block 1004 the method may include parsing thecorpus of data for the subject matter using a natural languageprocessing technique. The natural language processing technique may beconfigured to parse both structured data (e.g., tables, graphs) andunstructured data (e.g., textual content containing words, numbers,dates). In certain embodiments, the natural language processingtechnique may be a software tool, widget, or other program configured toanalyze and identify the semantic and syntactic elements andrelationships present in the corpus of data. More particularly, thenatural language processing technique can be configured to parse thegrammatical constituents, parts of speech, context, and otherrelationships (e.g., modifiers) of the corpus of data. The naturallanguage processing technique can be configured to recognize keywords,contextual information, and metadata tags associated with words,phrases, or sentences in the corpus of data. In certain embodiments, thenatural language processing technique can analyze summary information,keywords, figure captions, or text descriptions included in the corpusof data, and identify syntactic and semantic elements present in thisinformation. The syntactic and semantic elements can include informationsuch as word frequency, word meanings, text font, italics, hyperlinks,proper names, noun phrases, parts-of-speech, or the context ofsurrounding words. Other syntactic and semantic elements are alsopossible.

In certain embodiments, at block 1006 the method 1000 may includedetecting, based on the parsing, a first set of answers and a second setof answers. The first set of answers may include a first answerbelonging to a first answer category and a second set of answersbelonging to a second answer category. In certain embodiments, both thefirst and second answer categories may correspond to the subject matter.Generally, an answer (e.g., first answer, second answer) may refer to adata object or concept that may be returned in response to a query(e.g., a question in a question-answering system). In certainembodiments, the answer may correspond to a particular noun, entity,operation, or action. For example, in response to a question asking forthe name of the national bird, the answer may be returned as “baldeagle. In certain embodiments, the answer may correspond to an answercategory. The answer category may be a division or class of concepts orideas that include the answer. For instance, the answer of “bald eagle”may correspond to an answer category of “birds.” Additionally, eachanswer category may correspond to a subject matter. As described herein,the subject matter may be content or data related to particular topic,theme, or concept, and may include the answer category. As an example,referring to the example above, the answer category of “birds” may berelated to a subject matter of “animals,” “wildlife,” or the like.

As described herein, at block 1006 the method 1000 can include detectinga first set of answers and a second set of answers based on parsing acorpus of content related to a subject matter. In certain embodiments,the first and second set of answers may be detected by the naturallanguage processing system. For example, the natural language processingmay determine the words, phrases, or data present in the corpus thatcorresponds to the question received by a question answering system. Theanswers may be tagged or marked with an identifier to indicatecorrespondence to the question. As an example, in certain embodimentsthe question answering system may receive a question related totreatment options for a particular medical condition. The answers to thequestion may include a variety of medical treatments. The medicaltreatments may correspond to specific categories (e.g., answercategories) that represent a larger group of treatments. Morespecifically, the method 1000 may include detecting a first set ofanswers including a first answer of “antimetabolites” and a secondanswer of “cryosurgery.” The first answer may correspond to a firstanswer category of “chemotherapy,” and the second answer may correspondto a second answer category of “surgery.” Both the first and secondanswer categories may correspond to a subject matter such as “cancertreatments.” Other types of answers and answer categories are alsopossible.

Consistent with various embodiments, at block 1008 the method 1000 mayinclude identifying, based on the syntactic and semantic content, afirst set of ordering data for the first set of answers. The first setof ordering data may be structured or unstructured data or informationthat suggests (e.g., explicitly or implicitly) a particular order orsequence for the first answer and the second answer. The first set ofordering data may be identified using the syntactic content of thecorpus of data, the semantic content of the corpus of data, or both. Asan example, in certain embodiments, the ordering data may be a tablethat specifies a sequence of steps in which certain processes areperformed. In certain embodiments, the ordering data may be extractedfrom textual content of the corpus of data. For instance, the corpus ofdata may state a date or day of the week that a first step wasperformed, and another date or day of the week that a second step wasperformed. Using the included date or day of the week, the naturallanguage processing could determine the order of the first step and thesecond step. Additionally, the method 1000 may identify keywords such as“first,” “after,” “before,” “last,” and other words that may indicate atemporal order. As described herein, the natural language processingtechnique can be configured to identify the ordering data from bothunstructured and structured data environments.

As an example, consider the following paragraph, which may be a messageboard post returned in response to a query related to fixing a computer:

-   -   In order to fix my laptop, I had to reinstall the operating        system. Prior to that, however, I backed up all of my data to a        large external hard drive, and then proceeded to format the        internal hard drive of my laptop. Then I made a partition on the        freshly-formatted hard drive for the new OS. After restarting        the system and changing the boot priority to boot from the DVD        drive, I put in my OS CD, restarted the system again, and        followed the instructions to reinstall the operating system.        Then I replaced my backed-up data onto my laptop hard drive, and        before I knew it, it was operating like it was brand new.

As described herein, the method 1000 may detect answers including “Databackup,” “Hard drive format,” “Hard drive partition,” “System Restart,”“Change Boot Priority,” “OS CD Insertion,” “System Restart,” “OSInstallation Process,” and “Data Replacement.” At block 1008 the method1000 can identify ordering data in the form of temporal keywords such as“after,” “before,” “then,” “prior,” “proceeded to” as well as otherordering data that suggests a sequence for the detected answers. Incertain embodiments, the method 1000 can include marking the identifiedordering data with special tags or identifiers. For example, the method1000 may include highlighting the identified ordering data, or attachinga tag to each instance of ordering data. In certain embodiments, themethod 1000 may be configured to provide an ordering data reportindicating the identified ordering data in a particular corpus of data(e.g., it may be desirable for a user to see the factors that influencedthe order for a particular set of answers).

Consistent with various embodiments, at block 1010 the method 1000 caninclude determining, in response to identifying the first set ofordering data, a first answer sequence corresponding to an order of thefirst set of answers. The first answer sequence may be an arrangement,succession, or series of one or more answers (e.g., the first set ofanswers and the second set of answers). The arrangement of the answersin the first answer sequence may be associated with positive impacts(e.g., performance and efficiency benefits) in comparison to otherorders or configurations of the answers. As described herein, in certainembodiments, the first answer sequence may be determined using the firstset of ordering data identified for the first set of answers. Forinstance, referring to the example above, the identified ordering datasuch as the temporal keywords “after,” “before,” “then,” “prior,” and“proceeded to” may be used to determine a first answer sequence of “Harddrive format—Hard drive partition—System Restart—Change Boot Priority—OSCD Insertion—System Restart—OS Installation Process—Data Replacement.”

In certain embodiments, the method 1000 may include determining a secondanswer sequence. In certain embodiments, the second answer sequence maybe determined based on a second corpus of data different than the corpusof data used to identify the first answer sequence. In certainembodiments, the first and second answer sequences may be determinedusing the same corpus of data. More particularly, the method 1000 mayinclude detecting a third set of answers including a third answercorresponding to a third answer category, a fourth set of answersincluding a fourth answer corresponding to a fourth answer category, anda fifth set of answers including a fifth answer corresponding to a fifthanswer category. In certain embodiments, the third, fourth, and fifthanswer categories may relate to the subject matter. Based on syntacticand semantic content, the method 1000 may include identifying a secondset of ordering data for the third, fourth, and fifth sets of answers.In response to identifying the second set of ordering data, the method1000 may include determining a second answer sequence corresponding toan order of the third, fourth, and fifth sets of answers.

In certain embodiments, the method 1000 may include establishing asentiment factor for an answer sequence. The sentiment factor may be aninteger value between 1 and 100 that represents the relative sentiment(e.g., attitude, position, opinion, emotions) associated with an answersequence. As described herein, the sentiment factor for an answersequence may be determined based on an analysis of the contextualinformation, linguistic data, and semantic elements associated with aparticular answer sequence. As an example, an answer sequence thatincludes words and phrases such as “ineffective,” “poor performance,”and “problematic” may be characterized as having a substantiallynegative sentiment, while an answer sequence that is associated withwords and phrases such as “exceedingly efficient,” “effective” and“favorable outcome” may be characterized as having a substantiallypositive sentiment. As described herein, in certain embodiments, thenatural language processing technique may determine a sentiment factorfor the first and second answer sequence. The sentiment factor may be aninteger value that characterizes the attitude or emotions of the corpusof data with respect to the answer sequence. For instance, as describedherein, the sentiment factor may be an integer value between 1 and 100,wherein lower integers indicate a generally lower (e.g., substantiallynegative, or unfavorable) sentiment, and higher integers indicate agenerally higher (e.g., substantially positive, or favorable) sentiment.

In certain embodiments, the method 1000 may include comparing the firstanswer sequence and the second answer sequence based on the firstsentiment factor and the second sentiment factor. For example, considera scenario in which the first answer sequence has a first sentimentfactor of 76, and the second answer sequence has a second sentimentfactor of 53. In response to comparing the first and second sentimentfactors, the method 1000 may include rank-ordering (e.g., ranking,organizing, classifying) the first and second answer sequences based onthe comparison of the first and second sentiment factors. For instance,in certain embodiments, the method 1000 could include ranking the firstanswer sequence (e.g., the answer sequence with the greater sentimentfactor) above the second answer sequence (e.g., the answer sequence withthe lesser sentiment factor). Such an embodiment may provide benefitsassociated with identifying the answer sequence associated with the mostpositive results. Other methods of ranking the first and second answersequences are also possible.

FIG. 11 is a diagram illustrating an example system architecture 1100for managing answer sequences, consistent with embodiments of thepresent disclosure. Aspects of FIG. 11 are directed toward an answersequence discovery system for determining an answer sequence for one ormore answers, and using the discovered answer sequences to generateundiscovered answer sequences using an answer sequence module. As shownin FIG. 11, in certain embodiments, the example system architecture 1100can include an answer sequence discovery system 1102 and an answersequence generation system 1126. The answer sequence discovery system1102 can include a subject matter database 1104, an analysis component1104, a topic identification module 1106, a corpus selection module1108, a corpus parsing module 1110, a sentiment factor establishmentmodule 1112, a detection component 1113, a set of answers detectionmodule 1114, an answer category detection module 1116, an identificationcomponent 1117, an ordering data identification module 1118, an answersequence management component 1119, an answer sequence determinationmodule 1120, an answer sequence comparison module 1122, and an answersequence ranking module 1124. The answer sequence generation system 1126can include a rule management component 1127, an answer attributederivation module 1128, a rule definition module 1129 including acharacteristic identification sub-module 1130 and a rule establishmentsub-module 1132, an answer sequence model generation module 1134, a ruleaddition module 1136, a relationship extraction component 1137 includingan order component extraction module 1138 and an influence componentextraction module 1140, an answer sequence generation component 1141 andan answer combination module 1142.

Consistent with various embodiments, the analysis component 1104 maysubstantially correspond with the parsing block 1004 of FIG. 10. Incertain embodiments, the topic identification module 1106 can beconfigured to determine a topic of a question. The question may be aquery, statement, or other input received by a question answeringsystem. As described herein, the topic may be identified using naturallanguage processing techniques. Based on the identified topic of thequestion, the corpus selection module 1108 can be configured to select acorpus of data for a subject matter. In certain embodiments, the topicof the question may be related to the subject matter. As describedherein, the corpus parsing module 1110 may be configured to use anatural language processing technique configured to parse semantic andsyntactic content of the corpus of data. The sentiment factorestablishment module 1112 may be configured to use the semanticcharacteristics of the corpus of data to establish a quantitativeindication of the relative emotions or attitude associated with aparticular answer sequence.

Consistent with various embodiments, the detection component 1113 maysubstantially correspond with detecting block 1006 of FIG. 10. Incertain embodiments, the set of answers detection module 1114 may beconfigured to detect a first set of answers and a second set of answers(e.g., words, phrases, or data present in the corpus that corresponds tothe question) in response to the parsing of the corpus of data performedby the corpus parsing module 1110. Further, in certain embodiments, theanswer category detection module 1116 may be configured to detect answercategories (e.g., divisions or classes of concepts or ideas that includea respective set of answers) that correspond to the detected first andsecond set of answers.

Consistent with various embodiments, the identification component 1117may substantially correspond with identifying block 1008 of FIG. 10. Incertain embodiments, the ordering data identification module 1118 may beconfigured use the parsed semantic and syntactic content of the corpusof data to identify the ordering data. The ordering data may bestructured or unstructured data or information that suggests (e.g.,explicitly or implicitly) a particular order or sequence for the firstanswer and the second answer.

Consistent with various embodiments, the answer sequence managementcomponent 1119 may substantially correspond with the determining block1010 of FIG. 10. In certain embodiments, the answer sequencedetermination module 1120 may be configured to use the ordering dataidentified by the ordering data identification module 1118 to determinean answer sequence corresponding to an order of the first set ofanswers. The answer sequence may be an arrangement, succession, orseries of one or more answers (e.g., the first set of answers and thesecond set of answers). In certain embodiments, the answer sequencedetermination module 1120 may determine multiple answer sequencescorresponding to multiple sets of answers. Accordingly, in such anembodiment, the answer sequence comparison module 1122 can compare thedetermined answer sequences to one another. In certain embodiments, thedetermined answer sequences may be compared using a sentiment factorassociated with each answer sequence (e.g., the sentiment factorestablished by the sentiment factor establishment module 1112). Othermethods of comparing the answer sequences are also possible. In certainembodiments, the answer sequence ranking module 1124 can be configuredto rank-order the compared answer sequences. For example, the answersequence ranking module 1124 may rank-order the answer sequences basedon the sentiment factor associated with each answer sequence (e.g.,answer sequences with greater sentiment factors are ranked moreprominently). Other methods of rank-ordering the answer sequences arealso possible.

As described herein, certain embodiments of the present disclosure aredirected toward generating undiscovered answer sequences. In certainembodiments, generating the undiscovered answer sequences may includeusing an answer sequence module including a set of rules derived frompreviously discovered answer sequences. Accordingly, in certainembodiments, the system architecture 1100 can include an answer sequencegeneration system 1126. The answer sequence generation system caninclude components and modules configured to generate undiscoveredanswer sequences.

Consistent with various embodiments, the answer sequence generationsystem 1126 can include a rule management component 1127. The rulemanagement component 1127 may include modules and sub-modules directedtoward establishing rules to facilitate the generation of answersequences. In certain embodiments, the rule management component 1127may include an answer attribute derivation module 1128. The answerattribute derivation module 1128 may be configured to derive a set ofanswer attributes for a set of answers. In certain embodiments, theanswer attribute derivation module 1128 may derive a first set of answerattributes for a first set of answers, and a second set of answerattributes for a second set of answers. In certain embodiments, derivingthe set of answer attributes may include using the characteristicidentification module 1130 to identify a group of characteristics forthe set of answers that indicate a correspondence between a first answerand the second answer. Put differently, the set of answer attributes mayinclude particular traits or features that are distinctive of a specificanswer, and suggest a link between the answer and another answer.

For instance, consider an example in which a user wishes to makeadditional stock market investments. The set of answers detection modulemay detect a first answer of “PMJ Oil” and a second answer of “AKBEntertainment.” For the first answer of “PMJ Oil” the answer attributederivation module 1128 may derive a first answer attribute such as“Stock in oil companies is currently under-valued” and a second answerattribute of “Stock in broadcasting and entertainment companies iscurrently overvalued.” As described herein, in certain embodiments, theset of answer attributes may be derived from the semantic and syntacticcontent parsed by the natural language processing technique (e.g.,company financial statements, editorials of industry experts, and thelike.)

In certain embodiments, the rule establishment module 132 may beconfigured to establish rules (e.g. also referred to herein as answersequence rules) based on the derived attributes/identifiedcharacteristics for the first answer and the second answer. Generally,the rules may include principles, guidelines, facts, or indications thatcan be used to formalize the connection, link, or correspondence betweenthe first answer and the second answer. In certain embodiments, therules may define a procedure that describes a suggested means ofinteraction or sequential order for the first answer and the secondanswer. For instance, once again consider the example above, in whichthe first answer is “PMJ Oil,” and the second answer is “AKBEntertainment.” Based on the derived first answer attribute (e.g., Stockin oil companies is currently under-valued) and the second answerattribute (e.g., Stock in broadcasting and entertainment companies iscurrently overvalued) the rule establishment module 1134 may define arule (e.g., a first-second rule) such as “Stock in AKB Entertainmentshould not be purchased before stock in PMJ Oil” (e.g., it is a betterfinancial decision to buy undervalued stock while the price is low, andavoid buying stocks for which the price is overvalued.) In certainembodiments, the rule establishment module 1132 may be configured todefine multiple rules based on the derived attributes for the first andsecond answer. Although the present example was described in terms of afirst answer and a second answer, rules generated for situations withgreater or fewer answers are also possible.

Consistent with various embodiments, the answer sequence modelgeneration module 1134 may be configured to generate an answer sequencemodel for managing answer sequences. In certain embodiments, the answersequence model may be a database or other repository of answer sequencesand answer sequence rules. In certain embodiments, the answer sequencemodel may include using machine learning techniques configured toanalyze the answer sequences and answer sequence rules to inferrelationships, connections, and other links between various answers,answer categories, and answer sequences. For example, the answersequence model may include using inference algorithms to extract theconnections and links between different answer sequences. In certainembodiments, the links and connections extracted by the inferencealgorithms may be used to generate additional answer sequences (e.g.,undiscovered answer sequences.) In certain embodiments, the ruleaddition module 1136 may be configured to identify additional rules(e.g., based on a third set of answer attributes for a third answer anda fourth set of answer attributes for a fourth answer) and append themto the answer sequence model. For example, the rule addition module 1136may be configured to formalize the inferred connections and linksbetween two particular answers, and append them to the answer sequencemodel generation module 1134 in the form of additional rules.

Consistent with various embodiments, as described herein, therelationship extraction component 1137 may be configured to extractrelationships between two or more answer sequences to generateadditional answer sequences. Generally, the relationships may beinferred based on attributes or characteristics that are shared betweenmultiple answers or multiple answer sequences. In certain embodiments,the relationships may be formalized as higher-order rules (e.g., broaderthat the first-order answer sequence rules) or principles that governthe interactions between answers of different answer sequences. Incertain embodiments, extracting the relationship may include determiningan order component and an influence component of a given answer (e.g., afirst answer) with respect to another answer (e.g., a third answer). Incertain embodiments, the first answer and the third answer may belong toseparate answer sequences.

Generally, the order component may include an attribute orcharacteristic that suggests or governs (e.g., explicitly or implicitly)a particular order or sequence for the first answer with respect to thethird answer. For instance, the order component may suggest that thefirst answer occur before the third answer. In certain embodiments, theorder component may suggest that the first answer occur after the thirdanswer. The influence component may include an attribute orcharacteristic that indicates the degree of influence, impact, or effectthat a particular answer has on another answer. The influence componentmay, in certain embodiments, be expressed as an integer value between 0and 100, wherein higher numbers indicate substantially high influence,and lesser numbers indicate substantially little influence. For example,in certain situations, it may be very important that a certain answer inan answer sequence be accompanied by another answer (e.g., a certaintreatment must be followed by a particular medicine.) In certainsituations, it may be of relatively little importance whether aparticular answer is accompanied by another answer (e.g., whether or notsprinkles are included in a brownie recipe.) Accordingly, as describedherein, the answer sequence generation component 1141 may be configuredto generate an answer sequence using the first answer and the thirdanswer. In certain embodiments, generating the answer sequence mayinclude combining the first answer and the third answer based on theinfluence component and the order component.

FIG. 12 depicts an example of answer sequence generation 1200,consistent with various embodiments. Aspects of FIG. 12 are directedtoward generating undiscovered answer sequences using answer sequencerules defined for established answer sequences. More specifically, theexample of answer sequence generation 1200 illustrates an embodiment ofthe present invention directed toward oncology treatment plans. As shownin FIG. 12, the example of answer sequence generation 1200 may include aset of discovered answer sequences 1202 with a first answer sequence1210 and a second answer sequence 1220. The example of answer sequencegeneration 1200 may also include a set of generated answer sequences1222 with a third answer sequence 1230 and a fourth answer sequence1240. Each answer sequence may include a set of answers (e.g.,Chemotherapy C, Radiation B, etc., wherein Chemotherapy C and RadiationB are specific answers/treatment types within the respective answercategories of “chemotherapy” and “radiation.”)

As described herein, the present example may take place within aquestion-answering system environment. For example, as described herein,in response to a query of “What is the best way to treat cancer for apatient with the provided medical history?” the question-answeringsystem may determine, using a corpus of data including doctor's notes,medical journal articles, and research studies, that the treatment plansof the first answer sequence 1210 and the second answer sequence 1220are two known treatment plans for patients with the provided medicalhistory. Aspects of the present disclosure are directed toward using ananswer sequence generation model equipped with inference algorithms toanalyze the first answer sequence and the second answer sequence as wellas associated answer sequence rules, and extract relationships thatfacilitate the generation of additional answer sequences. For instance,based on semantic and syntactic information associated with the firstanswer sequence (e.g. past medical trials, medical history, oncologyjournals), a first answer sequence rule such as “Endocrine A may besafely followed by Radiation B,” for the first answer sequence.Similarly, a second answer sequence rule such as “Radiation B can befollowed by any type of surgery provided that Chemotherapy C is appliedimmediately afterwards.”

Accordingly, as described herein, the answer sequence model may beconfigured to analyze the first answer sequence rule and the secondanswer sequence rule, and extract a relationship between the firstanswer sequence and the second answer sequence in order to generateadditional answer sequences. For example, the answer sequence model maycombine the first answer sequence rule and the second answer sequencerule to deduce that, as Radiation B can safely be applied afterEndocrine A, and any type of surgery can be applied after Radiation B aslong as it is followed by Chemotherapy C, that the third answer sequence1230 and the fourth answer sequence 1240 are also possible. Accordingly,as described herein, the answer sequence model may generate the thirdanswer sequence 1230 and the fourth answer sequence 1240, and add themto a repository or database of known answer sequences.

Referring now to FIG. 13, a conceptual diagram illustrating a QA system1300 that classifies answers sorted according to answer category can beseen, according to embodiments of the present disclosure. The system1300 can include an answer sorter module 1304 and an answer classifiersystem 1310.

The answer sorter module 1304 can be the same or substantially similaras the answer sorter system 414 (FIG. 4). The answer sorter module 1304can be configured to sort answers generated in response to an inputquery into one or more answer categories. As described herein, theanswers can include corresponding answer confidence scores 1302 thatrepresent the QA system's 1300 confidence in each answer generated.

For example, the answer sorter module 1304 can be configured to sort afirst set of the answers into a first answer category and a second setof the answers into a second answer category. A set of answer categoryconfidence scores 1306 corresponding to the first set of answers can besorted into the first answer category. A set of answer categoryconfidence scores 1308 corresponding to the second set of answers can besorted into the second answer category.

The answer classifier system can be configured to manage confidence datain the QA system 1300. In embodiments the answer classifier system 1310can be configured to receive answer category confidence scores 1306,1308 as inputs. In embodiments, the answer classifier system can beconfigured to classify confidence scores in the answer categoryconfidence scores into one or more buckets, described further herein.For example, in FIG. 13, answer classifier 1310 can be seen receivinganswer category confidence scores 1308 as an input and outputting theconfidence scores sorted into one or more buckets 1312, 1314, 1316.Described further herein, the answer classifier 1310 can sort answercategory confidence scores using static thresholds and/or dynamicthresholds. Buckets 1312, 1314, 1316 can include one or more confidencescores labeled with a descriptions. For example, in FIG. 13 bucket 1312is labeled as “preferred”, bucket 1314 is labeled as “for consideration”and bucket 1316 is labeled as “not recommended”. In embodiments, answerclassifier 1310 can be configured to classify answers into buckets basedon the answer's corresponding confidence score.

Referring now to FIG. 14, a conceptual diagram illustrating a QA system1400 that classifies answers with buckets using multiple sets ofthresholds can be seen, according to embodiments of the presentdisclosure. In embodiments, some or all of the QA system 1400 can be anexample implementation of answer classifier 1310 (FIG. 13). FIG. 14depicts a QA system 1400 including an answer sorter module 1410, athreshold calculation module 1401, an answer quality module 1402, and ananswer grouper 1403. As described herein, the answer sorter module 1410can be configured to sort answers generated in response to an inputquery into one or more answer categories. As described herein, theanswers can include corresponding answer confidence scores 1404 thatrepresent the QA system's 1400 confidence in each answer generated.

As described herein, answers and the corresponding answer confidencescores can serve as an input to the answer sorter module 1410. Inembodiments, the answer sorter module 1410 can be the same orsubstantially similar as the answer sorter system 414 (FIG. 4). Theanswer sorter module 1410 can be configured to sort answers generated inresponse to an input query into one or more answer categories such asanswer category 1412. Answer category 1412 can be the same orsubstantially similar as described herein. Answer category 1412 caninclude a set of answers sorted into the answer category 1412 by theanswer sorter module 1410. The set of answers can include acorresponding set of answer category confidence scores 1404 representingthe QA system's confidence in each answer in the answer category 1412.

Answer confidence scores 1404 can serve as an input to the thresholdcalculation module 1401 and the answer quality module 1402. Thethreshold calculation module 1401 can be configured to calculatethresholds 1405 based on the answer confidence scores 1404. Inembodiments, the answer quality module 1402 classifies some of theanswer confidence scores 1404 with static thresholds and one or morebuckets. The answer confidence scores not classified with a bucket bythe answer quality module 1402 are unclassified answer confidence scores1407. For example, FIG. 14 depicts three buckets, a “preferred” bucket1406, a “for consideration” bucket 1409, and a “not recommended” bucket1408. The unclassified answer confidence scores 1407 and the calculatedthresholds 1405 serve as inputs into the answer grouper 1403.

The answer quality module 1402 and the threshold calculation module 1401can be configured to receive the answer confidence scores 1404. Thethreshold calculation module 1401 and the answer quality module 1402 canreceive the answer confidence scores 1404 in parallel or sequentially.In some instances, the answer quality module 1402 and the thresholdcalculation module 1401 receive the answer confidence scores 1404 from acomponent of the QA system 1400, such as an answer generator 328 (FIG.3) that generates the answer confidence scores 1404 and thecorresponding answers.

The answer quality module 1402 can be configured to classify answerconfidence scores 1404 with a “preferred” bucket 1406 and a “notrecommended” bucket 1408 based on static thresholds. Answer confidencescores not classified into the “preferred” bucket 1406 or into the “notrecommended” bucket 1408 are unclassified answer confidence scores 1407.For example, the answer quality module 1402 can apply the answer qualitythresholds of “0.9” and “0.1” for the “preferred” bucket 1406 and the“not recommended” bucket 1408, respectively. Therefore, in embodiments,answer confidence scores 1404 above a 0.9 can be placed into the“preferred” bucket 1406, and the answer confidence scores 1404 below 0.1can be placed into the “not recommended” bucket 1408. In embodiments,the static thresholds are determined before the answer confidence scores1404 are received. In embodiments, the static thresholds can allow auser to set answer quality thresholds that place certain answerconfidence scores into a particular bucket regardless of the value ofthe calculated thresholds 1405. For example, the static thresholds canoverride the calculated thresholds 1405, such that the static thresholdsprevent the calculated thresholds 1405 from removing some answerconfidence scores 1404 from the “preferred” bucket 1406 and/or the “notrecommended” bucket 1408.

The static thresholds can identify boundaries between buckets. In someembodiments, the static thresholds can be determined by anothercomponent of the QA system 1400. For example, a QA system componentcould monitor how often users select answers that fall outside of the“preferred” bucket 1406 and adjust the static thresholds accordingly.

The threshold calculation module 1401 can be configured to calculatethresholds 1405. The calculated thresholds 1405 can be calculated invarious ways. For example, to calculate the calculated thresholds 1405,the threshold calculation module 1401 can analyze the answer confidencescores 1404. In embodiments, the threshold calculation module 1401 canuse a data clustering technique, such as Jenk's natural breaksoptimization. In some embodiments, the threshold calculation module 1401can identify gaps and/or rates of changes associated with the answerconfidence scores, described further below. In embodiments, the numberof calculated thresholds 1405 is less than the number of buckets used(e.g., one calculated threshold per boundary between buckets). Forexample, in FIG. 14, a first threshold (0.88) is calculated thatdistinguishes the “preferred” bucket 1406 from the “for consideration”bucket 1409. A second threshold (0.42) is calculated that distinguishesbetween the “for consideration” bucket 1409 and the “not recommended”bucket 1408. Thus, because three buckets are used, two thresholds willbe calculated. These threshold values can be used by the answer grouper1403 to classify answers into buckets, described further herein.

In embodiments, the answer grouper 1403 applies the calculatedthresholds 1405 to the unclassified answer confidence scores 1407. Theanswer grouper 1403 can use the calculated thresholds 1405 to determinein which bucket an answer confidence score from the unassociated answerconfidence scores 1407 belongs. In embodiments, the answer grouper 1403compares each of the unassociated answer confidence scores 1407 to thelowest of the calculated thresholds 1405. Thus, the answer grouper 1403can associate the unassociated answer confidence scores 1407 that areless than the lowest of the calculated thresholds 1405 (0.42 in thisexample) with the “not recommended” bucket 1408. In embodiments, theanswer grouper 1403 then associates the still unassociated answerconfidence scores that are less than the next highest calculatedthreshold 1405 (0.88 in this example) with the “for consideration”bucket 1409. In embodiments, answer confidence scores left over areassociated with the “preferred” bucket 1406. In embodiments, the answerconfidence scores that the answer grouper 1403 associates with thebuckets are in addition to the answer confidence scores previouslyassociated with the buckets by the answer quality module 1402. Theanswer grouper 1403 can classify answer confidence scores into bucketswithout regard to the order of the answer confidence scores or the orderof the buckets. In embodiments, the answer grouper 1403 can usetechniques where answer confidence scores are associated into buckets inan order from least to greatest, from greatest to least, or in othervarious orders.

As described herein, the answer quality thresholds can override thecalculated thresholds 1405. For example, assume that the lower staticthresholds used by the answer quality module 1402 was “less than 0.5”.The answer quality module 1402 could associate the answer confidencescores 1404 of 0.43, 0.42, 0.15, 0.08, and 0.07 with the “notrecommended” bucket 108, despite the fact the answer grouper 1403 couldassociate values 0.43 and 0.42 with the “for consideration” bucket 1409based on the calculated thresholds 1405. In some instances, the QAsystem 1400 can have the calculated thresholds override the answerquality thresholds. For example, if all returned answers have an answerconfidence score in the range 0.9 to 1.0, the QA system 1400 couldselect to have the calculated thresholds override the answer qualitythresholds in order to prevent all returned answers from beingassociated with the “preferred” bucket 106.

Referring now to FIG. 15 a flow diagram illustrating example operationsfor associating answer category confidence scores with buckets can beseen, according to embodiments of the present disclosure. At operation1501, a number of buckets can be determined from configuration data. Inembodiments, there are at least two buckets. In some embodiments, thespecific number of buckets can vary. For example, it can be determinedbased on user experiments that a particular number of buckets is optimalfor a given scenario or set of scenarios (e.g., for questions from aparticular source).

In some instances, too many buckets can reduce the potential benefits ofbuckets. For example, if there was a bucket for each answer, the bucketsmight not generate an informative presentation of the answers. Further,system resources, such as processor speed and memory available mightimpose a practical limit on the number of buckets. The number of bucketsmight also be variable. For example, the number of buckets might changein proportion to the number of answers determined for a particularquery. Once the number of buckets has been determined, control can thenflow to operation 1502.

In some embodiments, more thresholds than buckets can be used to createa set of sub-buckets including one or more answer category confidencescores. In embodiments, the set of sub-buckets can then be distributedinto buckets according to a user distribution preference.

At operation 1502, a set of answer category confidence scores can bereceived. As described herein, the set of answer category confidencescores can be confidence scores corresponding to answers sorted into ananswer category. As described herein, each answer confidence score canbe associated with an answer. The answer confidence scores can bespecified in various manners. For example, the answer confidence scorescan be specified as percentages (or fractions of 100), integers within aparticular range, etc. After the answer confidence scores are received,control can then flow to operation 1504.

At operation 1504, it can be determined whether there are more answerconfidence scores than buckets. The number of buckets is the numberdetermined at operation 1501. In embodiments, the number of answerconfidence scores is equal to the number of answer confidence scoresreceived in operation 1502. In embodiments, if there are more answerconfidence scores than buckets, control can then flow to operation 1618in FIG. 16. If there are not more answer confidence scores than buckets,control can then flow to operation 1506.

In embodiments, at operation 1506, a loop in which each answerconfidence score is iterated over begins. The answer confidence scorecurrently being iterated over can be referred to hereinafter as the“selected answer confidence score”. In embodiments, during the firstpass through operation 1506, the selected answer confidence score isinitialized to a first answer confidence score. On each subsequent passthrough operation 1506, the selected answer confidence score can beupdated to be the next answer confidence score. In embodiments, the loopcontinues until all answer confidence scores have been iterated over. Inembodiments, after the selected answer confidence score has beeninitialized or updated, control can then flow to operation 1508.

In embodiments, at operation 1508, a nested loop in which a set ofstatic thresholds is iterated over begins. In embodiments, the staticthresholds are iterated over from least to greatest. The current staticthreshold currently being iterated over can be referred to hereinafteras the “selected static threshold”. The static thresholds can be used todistinguish one bucket from another bucket. As described herein, staticthresholds can be entered by a user, can be calculated based on thenumber of buckets, etc. In some embodiments, a different number ofbuckets than the number determined at operation 1501 can be used. Inembodiments, during an initial pass through operation 1508 afteroperation 1506, the selected static threshold can be initialized to thelowest static threshold. On each subsequent pass through operation 1508,the selected static threshold can be updated to be the next greateststatic threshold. In embodiments, the loop continues until the selectedanswer confidence score is less than the selected static threshold. Inembodiments, the loop will reinitialize on each iteration of the loopbeginning at operation 1506. After the selected static threshold hasbeen initialized or updated, control can then flow to operation 1510.

In embodiments, at operation 1510, it is determined whether the selectedanswer confidence score is less than the selected static threshold. Forexample, the selected answer confidence score is compared to theselected static threshold. If the answer confidence score is not lessthan the selected static threshold, control can then return to operation1508. In embodiments, if the answer confidence score is less than theselected static threshold, the nested loop is terminated and controlthen flows to operation 1512.

In embodiments, at operation 1512, the selected answer confidence scoreis associated with a bucket corresponding to the selected staticthreshold. For example, if the nested loop at operation 1508 wentthrough two iterations, then the selected answer confidence scorebecomes associated with a bucket corresponding to the second greateststatic threshold. An answer confidence score can be associated with abucket by inserting the answer confidence score or a pointer to theanswer confidence score into a data structure representing a bucket,inserting in a data structure representing the answer confidence score,an identifier for the associated bucket, etc. Once the selected answerconfidence score has been associated with the bucket, control can thenflow to operation 1516.

In embodiments, at operation 1516, it is determined whether there is anadditional answer confidence score. If there is an additional answerconfidence score that has not been associated with a bucket, control canthen return to operation 1506. In embodiments, if all answer confidencescores have been associated with a bucket, then the loop beginning at1506 terminates and the process ends.

Referring now to FIG. 16, a flow diagram illustrating example operationsfor associating answers with buckets can be seen, according toembodiments of the present disclosure. In embodiments, control flows tooperation 1618 if it was determined, at operation 1504 of FIG. 15, thatthere are more answer confidence scores than buckets.

In embodiments, at operation 1618, a clustering algorithm can be used todetermine dynamic thresholds. The dynamic thresholds can be determinedbased on the received answer confidence scores and can be different fordifferent sets of answer confidence scores. The dynamic thresholds canbe determined in a number of ways. For example, the dynamic thresholdscan be determined by using a data clustering technique, such as Jenk'snatural breaks optimization. In some examples, the dynamic thresholdscan be determined by using techniques that include identifying gapsand/or rates of changes associated with the answer confidence scores.

For example, the size of gaps between answer confidence intervals can beanalyzed for gaps over a certain threshold. The size of the gaps can becompared to the standard deviation of all of the gaps, for example.Additionally, the mean variance between answer confidence scores can becalculated, and the gaps can be compared to the mean variance. Theanswer confidence scores with gaps greater than or equal to the meanvariance or the standard deviation can be used as bucket thresholds. Insome embodiments, the dynamic thresholds can be determined bydetermining a plurality of gaps, each gap of the plurality of gapslocated between consecutive confidence scores of the confidence scores.Dynamic thresholds can be determined by determining a standard deviationassociated with the plurality of gaps and determining that a portion ofthe plurality of gaps is greater than or equal to the standarddeviation. In embodiments, the portion of the plurality of gaps asthresholds.

In some embodiments, dynamic thresholds can be determined by determininga plurality of rate changes. Each rate change of the plurality of ratechanges can be a rate change between consecutive confidence scores ofthe confidence scores. Dynamic thresholds can be determined bydetermining a portion of the plurality of rate changes to be a largestof the plurality of rate changes. In embodiments, the portion can beused as the dynamic threshold.

In embodiments, the dynamic thresholds are associated with buckets basedon the number of buckets and dynamic thresholds. In some embodiments,the dynamic thresholds can be used to define additional buckets.

In embodiments, at operation 1620, a loop in which each answerconfidence score is iterated over begins. In embodiments, at operation1622, a nested loop in which each static criterion is iterated overbegins. Answer quality criteria can allow answer confidence scores to beassociated with a specific bucket regardless of the other answerconfidence scores. Answer quality criteria can be generated by a moduleof the QA system. In some embodiments, it can be determined fromconfiguration data.

For example, configuration data could indicate that answer confidencescores below 0.3 should be placed in a “not preferred” bucket.Therefore, in embodiments, answer confidence scores less than 0.3 willbe placed in the “not preferred” bucket even if the answer confidencescore would be associated with a different bucket based on thethresholds determined in operation 1618.

In embodiments, the answer quality criteria can consist of numericalparameters such as ranges or greater than or less than values. In someembodiments, the answer quality criteria can be non-numericalparameters. For example, an answer, in addition to being associated withan answer confidence score, can be associated with other dataparameters, such as whether the answer is a known good answer, number oftimes the answer has been viewed, or amount of evidence supporting theanswer. An example of another static criterion is “answers that havebeen viewed more than 100 times.” Meeting such a criterion might result,for example, in an answer confidence score being placed in a “preferred”bucket. Additionally, for example, if an answer is a known good answer,it can automatically be placed in a “preferred” bucket, or, vice versa,a known bad answer in a “not preferred” bucket. Also, a static criterionmight be that if an answer is only supported by a small amount ofevidence, then it might be associated with a “for consideration” bucket.In embodiments, evidence that supports an answer can be text from adocument located in a corpus accessible by the QA system.

In embodiments, at operation 1624, it is determined whether the answerconfidence score meets the static criterion. If the answer confidencescore does not meet the static criterion, control then flows tooperation 1625. In embodiments, if the answer confidence score does meetthe static criterion, control then flows to operation 1626.

At operation 1625, it can be determined whether there is an additionalstatic criterion. If there is an additional static criterion, controlcan return to operation 1622. If each static criterion has been comparedto the selected answer confidence score, then the nested loop beginningat operation 1622 terminates and control can then flow to operation1628.

In embodiments, control can flow to operation 1626 if it was determined,at operation 1624, that the answer confidence score does meet the staticcriterion. At operation 1626, the answer confidence score can beassociated with a bucket corresponding to the static criterion. Ananswer confidence score can be associated with a bucket by inserting theanswer confidence score or a pointer to the answer confidence score intoa data structure representing a bucket. In some examples, associating ananswer confidence score with a bucket can include inserting anidentifier for the associated bucket in a data structure that indicatesthe answer confidence score. Once the answer confidence score has beenassociated with the bucket, control can then flow to operation 1628.

In embodiments, control flows to operation 1628 if it was determined, atoperation 1625, that there were no additional answer quality criteria.In embodiments, control also flowed to operation 1628 from operation1626. At operation 1628, it can be determined whether there is anadditional answer confidence score. In embodiments, if there is anadditional answer confidence score, then control returns to operation1620. If the answer confidence scores have been evaluated against theanswer quality criteria, then the loop beginning at 1620 terminates andcontrol can then flow to operation 1630.

In embodiments, at operation 1630, a loop in which each unassociatedanswer confidence score is iterated over begins. In embodiments, theunassociated answer confidence scores are those that were not associatedwith a bucket at operation 1626.

In embodiments, at operation 1632, a nested loop in which eachcalculated threshold is iterated over begins. The calculated thresholdscan be iterated over from least to greatest.

In embodiments, at operation 1634, it is determined whether theunassociated answer confidence score is less than the dynamic threshold.If the unassociated answer confidence score is not less than the dynamicthreshold, control can return to operation 1632. If the unassociatedanswer confidence score is less than the dynamic threshold, the nestedloop can be terminated and control then flows to operation 1636.

In embodiments, at operation 1636, the unassociated answer confidencescore is associated with a bucket corresponding to the dynamicthreshold. For example, if the nested loop at operation 1632 wentthrough two iterations, then the unclassified answer confidence score isassociated with a bucket corresponding to the second greatest dynamicthreshold. In embodiments, an unassociated answer confidence score canbe associated with a bucket by inserting the answer confidence score ora pointer to the answer confidence score into a data structurerepresenting a bucket, inserting in a data structure representing theanswer confidence score, an identifier for the associated bucket, etc.In embodiments, once the unassociated answer confidence score has beenassociated with the bucket, control then flows to operation 1638.

In embodiments, at operation 1638, it is determined whether there is anadditional unassociated answer confidence score. If there is anadditional unassociated answer confidence score that has not beencompared to the dynamic thresholds, control can then return to operation1630. In embodiments, if all unassociated answer confidence scores havebeen associated with a bucket, then the loop beginning at 1630terminates and the process ends.

Referring now to FIG. 17, a conceptual diagram illustrating a QA system1700 that distributes answers classified according to buckets can beseen, according to embodiments of the present disclosure. The system1700 can include an answer classifier 1704 and a bucket distributer1712.

Answer classifier 1704 can be configured to receive answer categoryconfidence scores 1702 as an input and output the confidence scoresclassified into buckets 1706, 1708, 1710. The buckets 1706, 1708, and1710 can be the same or substantially similar as described herein. Asdescribed herein, bucket 1706 could be labeled as a “preferred” bucket,bucket 1708 could be a “for consideration bucket”, and bucket 1710 couldbe a “not recommended” bucket. The answer classifier 1704 can beconfigured to classify confidence scores into one or more of the bucketsusing static thresholds and/or dynamic thresholds, as described herein.In embodiments, the answer classifier 1704 can be the same orsubstantially similar as described herein.

The bucket distributor 1712 can be configured to analyze the buckets1706, 1708, 1710 and distribute confidence scores among the bucketsbased on a preferred distribution of confidence scores. As describedherein, if too many confidence scores are placed within one or more ofthe buckets it can reduce the benefits of using buckets to organize theconfidence scores. Thus, the bucket distributor 1712 can be configuredto redistribute confidence scores among buckets based on the preferreddistribution of confidence scores.

In embodiments, the bucket distributor 1712 can be configured to receivethe buckets 1706, 1708, 1710 as inputs. The bucket distributor 1712 canbe configured to analyze each of the buckets 1706, 1708, 1710 todetermine a number of confidence scores sorted into each bucket. Thebucket distributor 1712 can be configured to determine whether a numberof confidence scores sorted into in one or more of the analyzed bucketsachieve a distribution threshold. The distribution threshold can be avalue representing the percentage of confidence scores in one bucketrelative to a total number of the answer category confidence scores1702. In embodiments, the number of confidence scores achieve thethreshold if the number of confidence scores exceeds the distributionthreshold. For example, in embodiments, the distribution threshold couldbe selected as 70%, so that if one of the buckets contains more than 70%of the total number of answer category confidence scores 1702, then thebucket achieves the distribution threshold.

For example bucket distributer 1712 could receive buckets 1706, 1708,and 1710 as an input and determine that bucket 1710 contains elevenconfidence scores out of a total of fifteen confidence scores. Thus,bucket distributer 1712 could determine that bucket 1710 contains 73% ofthe confidence scores and that bucket 1712 achieves a distributionthreshold of 70%.

The bucket distributor 1712 can then be configured to redistributeconfidence scores in the “large” bucket (the bucket that achieves thedistribution threshold) in response to determining that the number ofconfidence scores achieves the distribution threshold. In embodiments,the bucket distributor 1712 can be configured to perform clusteranalysis of the bucket to determine natural breaks within the bucket. Inembodiments, the bucket distributor 1712 can perform cluster analysis inthe same or substantially similar manner as described herein withreference to the answer classifier 1310 (FIG. 13). For example, inembodiments, bucket 1710 is broken into three sub-buckets 1714, 1716,and 1718 by the bucket distributor 1712.

In embodiments, the bucket distributor 1712 can then be configured toclassify the sub-buckets into one or more of the buckets 1706, 1708, and1710. In embodiments, the bucket distributor 1712 can be configured topromote, demote, or maintain confidence scores in the sub-buckets. Inembodiments, the bucket distributor can classify the sub-buckets basedon the bucket from which the sub-buckets were formed. In embodiments,the bucket distributor 1712 can move sub-buckets into buckets adjacentfrom the original bucket. For example, as sub-buckets 1714, 1716, and1718 were formed from the “not recommended” bucket 1710. Thus,sub-buckets can be promoted to the “for consideration” bucket 1708 ormaintained in the “not recommended” bucket 1710. In embodiments, thebucket distributor 1712 cannot remove all confidence scores from thelarge bucket. For example, in FIG. 17, some confidence scores must beretained in the “not recommended” bucket 1710. Thus, the bucketdistributor 1712 can be configured to maintain the third sub-bucket 1718in the “not recommended” bucket 1710.

In embodiments, the bucket distributor 1712 can be configured toclassify the sub-buckets into one or more of the buckets based on adistribution preference. In embodiments, the distribution preference canbe a user inputted preference as to which bucket is preferred forconfidence scores. For example, if could be preferred that moreconfidence scores should tend to be included in the “for consideration”bucket 1708 as answers in the “for consideration” bucket 1708 could bemore likely to be considered by a user than answers in the “notrecommended” bucket 1710.

The bucket distributor 1712 can classify sub-buckets into buckets basedon a number of confidence scores that would be in each bucket afterclassifying and the distribution preference. For example, the bucketdistributor 1712 could determine that classifying the second sub-bucket1716 with the third sub-bucket 1718 would result in the “notrecommended” bucket 1710 being larger than the “for consideration”bucket 1708. Further, the bucket distributor 1712 could determine thatclassifying the first and second sub-buckets 1714, 1716 into the “forconsideration” bucket 1708 would result in the “for consideration”bucket 1708 being larger than the “not recommended bucket”. Thus,because the bucket distributor 1712 has a distribution preference forthe “for consideration” bucket 1708, the bucket distributor 1712 wouldchoose to classify the first and second sub-buckets 1714, 1716 into the“for consideration” bucket 1708.

FIG. 18 is a flowchart illustrating a method 1800 for scoring answersequences, consistent with embodiments of the present disclosure.Aspects of FIG. 18 are directed toward determining a set of evaluationrules for a first answer sequence, and using the set of evaluation rulesto generate a sequence evaluation score for the first answer sequence.The method 1800 may begin at block 1802 and end at block 1812.Consistent with various embodiments, the method 1800 may include areceiving block 1804, an identifying block 1806, a determining block1808, and a generating block 1810.

Consistent with various embodiments, at block 1804 the method 1800 mayinclude receiving a set of answer sequences including a first answersequence. As described herein, an answer sequence may be an arrangement,succession, or series of one or more answers (e.g., the first set ofanswers). The arrangement of the answers in the first answer sequencemay be associated with positive impacts (e.g., performance andefficiency benefits) in comparison to other orders or configurations ofthe answers. In certain embodiments, the set of answer sequences may bereceived from a user via a visual user interface configured to receiveuser inputs. For example a user may manually enter a desired answersequence via the visual user interface, or select one of a set ofpossible answer sequences. In certain embodiments, the set of answersequences may be received via one of the methods or systems describedherein. For example, in certain embodiments, the method 1800 may receivethe set of answer sequences from the answer sequence discovery system1102 or the answer sequence generation system 1126 of FIG. 11. Incertain embodiments, the method 1800 may receive the set of answersequences in response to determining one or more answer sequences atblock 1010 of FIG. 10.

Consistent with various embodiments, at block 1806 the method 1800 caninclude identifying a set of scores coupled with the first set ofanswers. Generally, the set of scores can include data such as numbers,letters, or symbols that represent a quantitative indication of thequality, confidence, performance, success, or relevance of a particularanswer of the set of answers. For example, in certain embodiments, theset of scores can include confidence scores that represent thereliability of an answer or a set of answers in a question answeringsystem. As described herein, in certain embodiments, the set of scoresmay be coupled to the first set of answers. More particularly, eachanswer of the first set of answers may have an associated pre-determinedconfidence score. In certain embodiments, each answer may have multipleassociated scores (e.g., with conditions specifying the circumstances inwhich a certain score is to be used). Identifying the set of scores mayinclude using a natural language processing technique configured toparse structured and unstructured data associated with the first set ofanswers, and extracting the set of scores.

Consider the following example. In certain embodiments, the method 1800may, at block 1804, receive a first answer sequence. As describedherein, the first answer sequence may be associated with a subjectmatter, such as gardening. Further, in certain embodiments, the firstanswer sequence may include one or more answer categories. The answercategories may be divisions or classes of concepts or ideas that includeone or more answers of the first set of answers. The answer categoriesmay relate to the subject matter of the answer sequence. As an example,in certain embodiments, the first answer sequence may relate to asequence of steps for growing a bonsai tree. More particularly, thefirst answer sequence may include answer categories such as “Potting,”“Choosing a Location,” “Watering,” and “Feeding.” Within each answercategory may be a number of different answers, such as techniques andrecommended procedures for each step of the answer sequence. Forinstance, “Potting” may include answers such as “Pot in the spring,”“Pot when the buds extend,” and “Pot when the temperature is greaterthan 76 degrees Fahrenheit,” and “Watering” may include answers such as“Water when the top centimeter of soil is dry” and “Water when the rootsuncurl.” As described herein, each of the answers may have an associatedscore (e.g. confidence value) that represents the reliability of theanswer. In certain embodiments the score may be an integer between 1 and100, where lower numbers are associated with relatively littlereliability, and higher numbers are associated with relatively greaterreliability. For instance, the answer of “Pot in the spring,” may beassociated with a score of 84, “Pot when the buds extend,” may beassociated with a score of 64, and “Pot when the temperature is greaterthan 76 degrees Fahrenheit” may be associated with a score of 47.Similarly, “Water when the top centimeter of soil is dry” may beassociated with a score of 89, and “Water when the roots uncurl” may beassociated with a confidence score of 39.

Consistent with various embodiments, at block 1808 the method 1800 mayinclude determining, based on a subject matter corresponding to thefirst answer sequence, a set of evaluation rules. As described herein,in certain embodiments, the first answer sequence may correspond to asubject matter. The subject matter may include content or data relatedto a particular topic, theme, or concept. As examples, the subjectmatter may relate to 19^(th) century literature, semiconductors, haiku,or woodworking. The set of evaluation rules may be a group ofestablished principles, guidelines, or regulations that can be used toassess the set of answers of a particular answer sequence, and determinean overall answer sequence evaluation score for the first answersequence.

In certain embodiments, determining the set of evaluation rules togenerate the sequence evaluation score may be based on the subjectmatter of the first answer sequence. More particularly, at block 1808the method 1800 may include selecting one or more sets of evaluationrules based on characteristics of the subject matter that suggest that acertain set of evaluation rules is suitable. For instance, aspects ofthe present disclosure relate to the recognition that, in certainsituations, there may be benefits associated with evaluating an answersequence for a first subject matter with particular caution (e.g.,medical treatments, oncology, investment plans), while answer sequencesfor other subject matters (baking brownies, sewing scarves) may not needto be evaluated with the same degree of caution. Further, in certainsituations, a particular set of evaluation rules may be desirable inscenarios when certain pertinent information regarding the subjectmatter is available. Accordingly, aspects of the present disclosure aredirected toward determining the set of evaluation rules based oncharacteristics of the subject matter.

Accordingly, in certain embodiments, determining the set of evaluationrules may include computing a caution value for the first answersequence. In certain embodiments, the caution value may be based on thesubject matter. Generally, the caution value may be a quantitativeindication of the seriousness, potential for risk, or severityassociated with a particular subject matter. In certain embodiments, thecaution value may be an integer between 1 and 100, wherein lower numbersindicate a lower degree of caution and higher numbers indicate a greaterdegree of caution. As described herein, in certain embodiments thecaution value may be computed using a natural language processingtechnique configured to parse semantic and syntactic content associatedwith the first answer sequence. For instance, in certain embodiments,the natural language processing technique may be configured to parse acorpus of subject matter data relating to the first answer sequence. Incertain embodiments, computing the caution value may include using thenatural language processing technique to identify words that indicatethat a particular degree of caution be used when considering a givenanswer sequence (e.g., “risk,” “danger,” “accident,” “careful,” “heed,”“surgery,” “injury,” “serious,” “threat,” “hazard,” “cancer.”) Further,in certain embodiments, the method 1800 may include comparing semanticcontent for the first answer sequence with an ontology framework ofstructured relationships in order to identify particular subject mattersthat have been flagged as “serious” (e.g., oncology, surgery,investments, severe weather). Other methods of computing the cautionvalue are also possible.

In certain embodiments, in response to computing the caution value forthe first answer sequence, the method 1800 may include comparing thecaution value to a first caution threshold. The first caution thresholdmay be a predetermined caution value that, when exceeded, prompts theselection of a first evaluation rule. As an example, in certainembodiments, the first caution threshold may be 64. Accordingly, a firstanswer sequence with a computed caution value of 67 achieves the firstcaution threshold of 64, and may prompt selection of the firstevaluation rule.

In certain embodiments, the first evaluation rule may includeidentifying a first score of the set of scores coupled with the firstset of answers. In certain embodiments, the first score may not achieve(e.g., be below) a first score threshold. As an example, in certainembodiments, the scores associated with the first set of answers may bedistributed into score quintiles, with each quintile representing 20% ofthe score range associated with the first set of answers. For instance,for a first answer sequence having four answers with scores of 0, 34,51, and 100, score quintiles may be created to cover score ranges from1-20, 21-40, 41-60, 61-80, and 81-100. In certain embodiments, the firstscore threshold may be a value corresponding to 20% of the lowestquintile. In certain embodiments, the first score threshold may be 1% ofthe lowest quintile. Other methods of setting the first score thresholdare also possible. In certain embodiments, the first score threshold maybe 5% greater than the lowest score included in the first set ofanswers. Accordingly, as described herein, at block 1810 the method 1800can include assigning the first score to the first answer sequence asthe sequence evaluation score. Aspects of the present disclosure, incertain embodiments, are directed toward selecting the lowest score ofthe set of scores, and assigning it to the first answer sequence as thesequence evaluation score. Such a configuration may be associated withbenefits such as providing (e.g., to a user) a conservative outlook forthe first answer sequence.

As described herein, aspects of the present disclosure are directedtoward selecting a second of evaluation rule to evaluate the firstanswer sequence. In certain embodiments, the second evaluation rule maybe selected in response to determining that the caution value for thefirst answer sequence does not achieve a second caution threshold. Asdescribed herein, the second caution threshold may be a predeterminedcaution value that, when exceeded, prompts the selection of the secondevaluation rule. In certain embodiments, the second confidence thresholdmay be equal to the first confidence threshold. Determining that thecaution value does not achieve the second caution threshold may includecomparing the caution value to the second caution threshold. As anexample, in a situation where the second caution threshold is 71, acaution value of 44 may fail to achieve the second caution threshold.

In response to determining that the caution value does not achieve thesecond caution threshold, the method 1800 may include selecting a secondevaluation rule. In certain embodiments, the second evaluation rule mayinclude calculating an aggregate score for the first answer sequencebased on the first set of scores. Generally, the aggregate score may bea cumulative or composite score generated using the first set of scores.For instance, the aggregate score may be calculated using a statisticalalgorithm such as contra-harmonic mean algorithms, quadratic meanalgorithms, arithmetic mean algorithms, geometric mean algorithms, andthe like. As a basic example, for a first answer sequence with a set ofscores including 38, 27, 95, and 74, the method 1800 may include using acontra-harmonic mean algorithm to generate an aggregate score of 71.3for the first answer sequence. In embodiments, the method 1800 mayinclude calculating an arithmetic mean of 58.5. Other algorithms andother methods of calculating the aggregate score are also possible.Accordingly, as described herein, aspects of the present disclosure aredirected toward calculating the aggregate score and assigning it to thefirst answer sequence as the sequence evaluation score (e.g., at block1810 of method 1800). Such a configuration may be associated withbenefits such as providing an inclusive, overall summary of thereliability of the first answer sequence.

In certain embodiments, aspects of the present disclosure are directedtoward selecting a third evaluation rule to evaluate the first answersequence. Aspects of the third evaluation rule are directed towardproviding a comprehensive, refined evaluation of the first answersequence. Accordingly, in certain embodiments, aspects of the presentdisclosure are directed toward identifying a set of answer categoriescorresponding to the set of answers of the first answer sequence. Asdescribed herein, the set of answer categories may be divisions orclasses of concepts or ideas that include one or more answers of a setof answers. In certain embodiments, the set of answer categories mayrelate to a subject matter of an answer sequence. For example, for asubject matter of “oncology,” the method 1800 may include identifyinganswer categories of “endocrine,” “chemotherapy,” “radiation,” and“surgery.” As described herein, the answer categories may be identifiedusing a natural language processing technique, and substantiallycorrespond to block 1006 of FIG. 11.

In certain embodiments, in response to identifying the set of categoriescorresponding to the set of answers of the first answer sequence, themethod 1800 may include collecting context data for the set ofcategories. The context data may indicate a relative importance of afirst answer category of the set of answer categories to the firstanswer sequence as a whole. The context data may also indicate therelative importance of the first answer category in relation to theother answer categories of the set of answer categories. Generally, thecontext data may include a corpus of textual, video, audio, or otherdata that provides information relating to the background and additionalexplanation, elaboration, or details regarding a particular answercategory. As an example, once again referring to the example aboveregarding growing bonsai trees with answer categories of “Potting,”“Choosing a Location,” “Watering,” and “Feeding,” the method 1800 mayinclude identifying context information such as bonsai growing guides,journal articles in botanical magazines, and user created video contentpertaining to bonsai trees. Other types of context data are alsopossible.

In certain embodiments, the method 1800 may include evaluating thecollected context data. For instance, in certain embodiments, thecontext data may be evaluated by using a natural language processingtechnique configured to parse semantic and syntactic content of thecontext data. Evaluating the context data may include assessing thecontent of the context data, and ascertaining the usefulness of thecontext data with respect to the first answer sequence. Moreparticularly, evaluating the context data can include determining thatthe context data achieves a satisfaction criterion. The satisfactioncriterion may, in certain embodiments, be a standard or benchmark togauge the relative quality or relevance of the collected context data.For instance, satisfaction criterion, in certain embodiments, mayinclude a stipulation that the context data include mention of arelation to either the subject matter of the answer sequence, anotheranswer category of the answer sequence, or both in order to achieve thesatisfaction criterion. Accordingly, a journal article (e.g., contextdata) that includes a sentence such as “It is agreed upon by mostexperts that careful watering techniques are the single most importantfactor in raising a healthy bonsai” may be determined to achieve thesatisfaction criterion (e.g., mention of “raising a healthy bonsai” issubstantially similar to the subject matter of the answer sequence.)Additionally, a journal article that includes a sentence such as “Whileimportant, potting and repotting a bonsai is not as crucial to thehealth of a bonsai as is choosing a suitable location for it,” may alsobe determined to achieve the satisfaction criterion (e.g., a relationbetween the answer categories of “potting” and “choosing a location” wasmentioned.) Other possible satisfaction criteria are also possible.

Accordingly, aspects of the present disclosure, in certain embodiments,are directed toward selecting a third evaluation rule in response todetermining that the context data achieves the satisfaction criterion.As described herein, aspects of the third evaluation rule may bedirected toward providing a comprehensive evaluation of the first answersequence by making use of the context data for each answer category. Incertain embodiments, the third evaluation rule may include assigning,based on the context data, a weighting value to each answer category ofthe set of answer categories. For instance, for an answer sequencehaving two answer categories, the third rule may include assigning afirst weighting value to a first answer category and a second weightingvalue to the second answer category. Generally, the weighting value maybe a factor that provides a quantitative representation of themagnitude, impact, or significance of a particular category in relationto the other answer categories of the answer sequence or the answersequence as a whole. The weighting value may be assigned to eachcategory of the answer categories using information that was present inthe context data. The weighting value may, in certain embodiments, be aninteger between 0 and 10. For example, referring once again to theexample above related to growing a bonsai tree, the answer category of“Potting” may be assigned a weighting value of 4, and the answercategory of “Choosing a Location” may be assigned a weighting value of 7(e.g., the context data indicated that the answer category of “Choosinga Location” was more significant than was the answer category of“Potting.”)

Accordingly, in response to assigning weighting values to each answercategory of an answer sequence, the method 1800 may include calculatingan aggregate score for the answer sequence using the individualweighting values for each respective answer category. As describedherein, calculating the aggregate score for the answer sequence may usea statistical algorithm or other technique, such as a contra-harmonicmean technique, a geometric-arithmetic mean technique, or the like.

Consistent with various embodiments, at block 1810 the method 1800 caninclude generating, based on the set of scores and the set of evaluationrules, a sequence evaluation score for the first answer sequence. Asdescribed herein, the sequence evaluation score may represent an overallassessment of the reliability or confidence of the first answersequence, and may be calculated and assigned to the first answersequence using one or more of a set of evaluation rules. Althoughreference is made herein to selecting a particular evaluation rule,embodiments combining multiple evaluation rules, including those notdisclosed explicitly herein, are also possible.

FIG. 19 is a high level flow-diagram of a method 1900 for scoring answersequences, according to embodiments. Aspects of FIG. 19 are directedtoward determining (e.g., selecting) an evaluation rule, and using it tocalculate and assign a sequence evaluation score to a first answersequence. As shown in FIG. 19, the method 1900 may, at block 1902,receive answer sequences. Receiving the answer sequences at block 1902may substantially correspond with receiving block 1804 of the method1800. At block 1904 the method 1900 may include identifying a set ofscores (e.g., confidence values) for an answer sequence. Identifying theset of scores at block 1904 may substantially correspond with block 1806of the method 1800.

At block 1906, aspects of the present disclosure are directed towarddetermining an evaluation rule. As described herein, determining theevaluation rule for a particular answer sequence may depend on thecharacteristics of the subject matter and the information availableregarding the answer sequence and the answer categories it includes. Incertain embodiments, when a substantial amount of information regardingthe answer sequence and the answer categories are available, the thirdevaluation rule may be chosen. In embodiments where less informationregarding the answer sequence and the answer categories are available,the first or second evaluation rules may be chosen. Combinations of theevaluation rules, as well as other evaluations rules, are also possible.

At block 1908, aspects of the present disclosure are directed towardcomputing a caution value based on the subject matter for the answersequence. The caution value may be a quantitative indication of theseriousness, potential for risk, or severity associated with aparticular subject matter. At the caution threshold decision block 1910,the caution value may be compared to a caution threshold. If the cautionvalue is greater than the caution threshold, the first evaluation rulemay be selected at block 1912. If the caution value is less than thecaution threshold, the second evaluation rule may be selected at block1916.

As described herein, in response to selecting the first evaluation ruleat block 1912, at block 1914 aspects of the present disclosure aredirected toward applying the first evaluation rule and identifying afirst score of the set of scores associated with the set of answers ofthe answer sequence. In certain embodiments, the first score may bebelow a first score threshold. In embodiments, the first score may bethe lowest score of the set of scores. Accordingly, in response toselecting the first score, at block 1932 the first score may be assignedto the answer sequence.

As described herein, if the caution value does not achieve the cautionvalue threshold, aspects of the present disclosure are directed towardselecting the second evaluation rule at block 1916. At block 1918, thesecond evaluation rule may be applied, and an aggregate score may becalculated for the answer sequence. The aggregate score may be acumulative or composite score generated using the first set of scores.For instance, the aggregate score may be calculated using anarithmetic-geometric mean technique, arithmetic mean-technique,contra-harmonic mean technique, or other statistical algorithm using thefirst set of scores. Accordingly, in response to calculating theaggregate score, at block 1932 the aggregate score may be assigned tothe answer sequence.

At block 1920, aspects of the present disclosure are directed towardidentifying and filtering a set of answer categories for the answersequence. The answer categories may be divisions or classes of conceptsor ideas that include one or more answers of the first set of answers.The answer categories may relate to the subject matter of the answersequence. In certain embodiments, the answer categories may be filteredfrom the answer sequence. For instance, at block 1920 the score of eachanswer of the set of answer categories may be compared to a scorethreshold, and answer categories that do not include an answer thatachieves the score threshold may be removed from the answer sequence.Accordingly, such a configuration may be associated with benefits suchas providing reliable and confident answer sequences (e.g., a pooranswer or answer category may drag down an otherwise good answersequence.)

At block 1922, aspects of the present disclosure are directed towardcollecting and evaluating context data for the set of answer categories.The context data may be textual, audio, video, or other content thatindicates a relative importance of the first answer category in relationto the other answer categories of the set of answer categories or theanswer sequence as a whole. The context data may be collected from acorpus of data such as a digital encyclopedia, journal articles,research results, studies, and the like. The context data may beevaluated using a natural language processing technique configured toparse semantic and syntactic content of the context data. At block 1924,aspects of the present disclosure are directed toward determiningwhether the context data achieves a satisfaction criterion. Thesatisfaction criterion may be a standard or benchmark to gauge therelative quality or relevance of the collected context data.

As described herein, in response to determining that the context dataachieves the satisfaction criterion, the third evaluation rule may beselected at block 1926. Aspects of the third evaluation rule may bedirected toward assigning weighting values to each answer category ofthe answer sequence, and calculating an aggregate score for the answersequence using the weighting values. Accordingly, at block 1928, thethird evaluation rule may be applied and weighting values may beassigned to each answer category based on the context data collected atblock 1922.

At block 1930, aspects of the present disclosure are directed towardadjusting the weighting value assigned to each answer category of theanswer sequence. In certain embodiments, adjusting the weighting valueassigned to each category may include receiving a first set of answerpreference data from a user. The answer preference data may indicate aninclination or a disinclination (e.g., of a user) for a particularanswer or answer category of the answer sequence. Accordingly, based onthe first set of answer preference data, at block 1930 the weightingvalues assigned to the answer categories may be adjusted. For instance,consider an example related to cancer treatment, in which an individualhas a strong objection to chemotherapy. Accordingly, the weighting valueassigned to the answer category of chemotherapy may be decreased.Similarly, for an example related to investment options, an individualmay have a strong predilection for long-term savings. Accordingly, theweighting value assigned to an answer category of “savings bonds” may beincreased. Other methods of adjusting the weighting values are alsopossible.

At block 1932, aspects of the present disclosure are directed towardassigning a sequence evaluation score to an answer sequence. Asdescribed herein, the sequence evaluation score may be the first scoreidentified at block 1914 based on the first evaluation rule, theaggregate score calculated at block 1918 based on the second evaluationrule, calculated at block 1932 using the weighting values assigned basedon the third evaluation rule, or generated using another method. Asdescribed herein, the sequence evaluation score may represent an overallassessment of the reliability or confidence of the first answersequence.

In certain embodiments, at block 1932, aspects of the present disclosureare directed toward modifying the sequence evaluation score of a firstanswer sequence based on a comparison with a second answer sequence. Putdifferently, the reliability of an answer sequence may be judgedrelative to the contents of other answer sequences (e.g., an answersequence that fails to include an important answer category may bepenalized.) Accordingly, in certain embodiments, aspects of the presentdisclosure are directed toward comparing an answer sequence (e.g., afirst answer sequence) with another answer sequence (e.g., a secondanswer sequence), and identifying a first answer category that belongsto the first answer sequence but is absent from the second answersequence. In response to identifying the first answer category, it maybe determined that a first score coupled with a first answer of thefirst category achieves a first influence threshold. Generally, thefirst influence threshold may be a quantitative indication of the degreeto which the first answer category impacts the sequence evaluationscore. In response to determining that the first score achieves thefirst influence threshold, the sequence evaluation score of the secondanswer sequence may be modified. Modifying the sequence evaluation scoreof the second answer sequence may include increasing, decreasing, orotherwise adjusting the sequence evaluation score of the second answersequence.

For instance, consider once more the example above pertaining to raisinga bonsai tree. As described herein, a first answer sequence may includeanswer categories of “Potting,” “Choosing a Location,” “Watering,” and“Feeding.” A second answer sequence may include answer categories of“Potting,” “Choosing a Location,” and “Feeding.” Accordingly, aspects ofthe present disclosure are directed toward comparing the first answersequence with the second answer sequence, and determining that theanswer category of “Watering” is included in the first answer sequencebut not the second answer sequence. Further, the identified answercategory may be evaluated to determine whether a first score coupledwith a first answer achieves a first influence threshold. In certainembodiments, the first influence threshold may be 85. Accordingly, ananswer of “Water when the top centimeter of soil is dry” with a firstscore of 89 may be determined to achieve the influence threshold. As thesecond answer sequence does not include the answer category of“Watering,” which includes a substantially significant answer, thesequence evaluation score of the second answer sequence may bedecreased. In certain embodiments, the magnitude of the decrease may beproportional to the first score of the first answer (e.g., the greaterthe significance of the missing answer, the greater the second answersequence is penalized.) In certain embodiments, aspects of the presentdisclosure are directed toward using a placeholder null value (e.g., 0)in place of the missing answer category during calculation of thesequence evaluation score. Other methods of modifying the sequenceevaluation score of the second answer sequence are also possible.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present disclosurehave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A method for managing confidence data in aquestion-answering environment, the method comprising: sorting, based ona set of answer categories for a subject matter, a first set of aplurality of answers into a first answer category and a second set ofthe plurality of answers into a second answer category, each of thefirst set corresponding to at least one of a third set of a plurality ofconfidence scores and each of the second set corresponding to at leastone of a fourth set of the plurality of confidence scores, the pluralityof confidence scores representing confidence of answers to a querysubmitted to a question-answering system; classifying confidence scoresof the third set into one of a plurality of confidence buckets using afirst threshold; determining a fifth set of a plurality of thresholdsusing the plurality of confidence scores; and classifying unclassifiedconfidence scores of the third set into one of the plurality ofconfidence buckets using the fifth set of the plurality of thresholds.2. The method of claim 1, further comprising: classifying confidencescores of the fourth set into one of the plurality of confidence bucketsusing the first threshold; determining a sixth set of a plurality ofthresholds using the plurality of confidence scores; and classifyingunclassified confidence scores of the fourth set into one of theplurality of confidence buckets using the sixth set of the plurality ofthresholds.
 3. The method of claim 1, further comprising: presenting,via the question-answering system, the first set of the plurality ofanswers corresponding to the third set of the plurality of confidencescores, in accordance with the classifying of the third set of theplurality of confidence scores into the plurality of confidence buckets.4. The method of claim 1, wherein: the first threshold is a staticthreshold, and wherein classifying the third set of the plurality ofconfidence scores into one of the plurality of confidence buckets inaccordance with the first threshold includes: determining that a portionof the third set of the plurality of confidence achieves the staticthreshold; and classifying the portion of the third set of the pluralityof confidence scores into one of the plurality of confidence buckets inresponse to determining that the portion of the third set achieves thestatic threshold.
 5. The method of claim 1, wherein: determining thefifth set of the plurality of thresholds using the plurality ofconfidence scores includes: determining the fifth set of the pluralityof thresholds using the third set of the plurality of confidence scores.6. The method of claim 5, wherein: determining the fifth set of aplurality of thresholds using the third set of the plurality ofconfidence scores includes: determining a plurality of gaps, each gap ofthe plurality of gaps located between consecutive confidence scores ofthe third set of the plurality of confidence scores; determining astandard deviation associated with the plurality of gaps; determiningthat a portion of the plurality of gaps is greater than or equal to thestandard deviation; and using the portion of the plurality of gaps asthe fifth set of the plurality of thresholds in response to determiningthat the portion of the plurality of gaps is greater than or equal tothe standard deviation.
 7. The method of claim 5, wherein: determiningthe fifth set of the plurality of thresholds using the third set of theplurality of confidence scores includes: determining a plurality of ratechanges, wherein each rate change of the plurality of rate changes is arate change between consecutive confidence scores of the third set ofthe plurality of confidence scores; determining a portion of theplurality of rate changes to be a largest of the plurality of ratechanges; and using the portion of the plurality of rate changes as thefifth set of the plurality of thresholds.
 8. The method of claim 1,further comprising: determining that one of the plurality of confidencebuckets includes a number of confidence scores that achieves a secondthreshold; determining a sixth set of a plurality of thresholds usingthe number of confidence scores; and classifying a portion of the numberof confidence scores into one of the plurality of confidence bucketsusing the sixth set of the plurality of thresholds.